TY - JOUR
T1 - An explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in women with breast cancer
AU - Gao, Yuan
AU - Ventura-Diaz, Sofia
AU - Wang, Xin
AU - He, Muzhen
AU - Xu, Zeyan
AU - Weir, Arlene
AU - Zhou, Hong Yu
AU - Zhang, Tianyu
AU - van Duijnhoven, Frederieke H.
AU - Han, Luyi
AU - Li, Xiaomei
AU - D’Angelo, Anna
AU - Longo, Valentina
AU - Liu, Zaiyi
AU - Teuwen, Jonas
AU - Kok, Marleen
AU - Beets-Tan, Regina
AU - Horlings, Hugo M.
AU - Tan, Tao
AU - Mann, Ritse
N1 - Funding Information:
We acknowledge and are grateful to the six participating breast radiologists for involving in the reader study of NAT response prediction. Our thanks also to Sofia Ventura for designing Fig. . Laura Estacio, George Agrotis, Luuk Balkenende for fruitful discussions, ideas, supporting analysis, and engineering that either directly or indirectly made this work possible. The authors thank Grand Challenge for offering the reader study platform. The authors thank the support of the China Scholarship Council (202107720016, 202006930001, and 202006240065), Guangzhou Elite Project (TZ-JY201948), Macao Polytechnic University Grant (RP/FCA-15/2022) and Science and Technology Development Fund of Macao (0105/2022/A).
Publisher Copyright:
© The Author(s) 2024.
PY - 2024/12/1
Y1 - 2024/12/1
N2 - Multi-modal image analysis using deep learning (DL) lays the foundation for neoadjuvant treatment (NAT) response monitoring. However, existing methods prioritize extracting multi-modal features to enhance predictive performance, with limited consideration on real-world clinical applicability, particularly in longitudinal NAT scenarios with multi-modal data. Here, we propose the Multi-modal Response Prediction (MRP) system, designed to mimic real-world physician assessments of NAT responses in breast cancer. To enhance feasibility, MRP integrates cross-modal knowledge mining and temporal information embedding strategy to handle missing modalities and remain less affected by different NAT settings. We validated MRP through multi-center studies and multinational reader studies. MRP exhibited comparable robustness to breast radiologists, outperforming humans in predicting pathological complete response in the Pre-NAT phase (?AUROC 14% and 10% on in-house and external datasets, respectively). Furthermore, we assessed MRP’s clinical utility impact on treatment decision-making. MRP may have profound implications for enrolment into NAT trials and determining surgery extensiveness.
AB - Multi-modal image analysis using deep learning (DL) lays the foundation for neoadjuvant treatment (NAT) response monitoring. However, existing methods prioritize extracting multi-modal features to enhance predictive performance, with limited consideration on real-world clinical applicability, particularly in longitudinal NAT scenarios with multi-modal data. Here, we propose the Multi-modal Response Prediction (MRP) system, designed to mimic real-world physician assessments of NAT responses in breast cancer. To enhance feasibility, MRP integrates cross-modal knowledge mining and temporal information embedding strategy to handle missing modalities and remain less affected by different NAT settings. We validated MRP through multi-center studies and multinational reader studies. MRP exhibited comparable robustness to breast radiologists, outperforming humans in predicting pathological complete response in the Pre-NAT phase (?AUROC 14% and 10% on in-house and external datasets, respectively). Furthermore, we assessed MRP’s clinical utility impact on treatment decision-making. MRP may have profound implications for enrolment into NAT trials and determining surgery extensiveness.
U2 - 10.1038/s41467-024-53450-8
DO - 10.1038/s41467-024-53450-8
M3 - Article
SN - 2041-1723
VL - 15
JO - Nature Communications
JF - Nature Communications
IS - 1
M1 - 9613
ER -