Multimodal Breast MRI Language-Image Pretraining (MLIP): An Exploration of a Breast MRI Foundation Model

Nika Rasoolzadeh; Tianyu Zhang; Yuan Gao; Jarek M. van Dijk; Qiuhui Yang; Tao Tan; Ritse M. Mann

doi:10.1007/978-3-031-77789-9_5

Multimodal Breast MRI Language-Image Pretraining (MLIP): An Exploration of a Breast MRI Foundation Model

Nika Rasoolzadeh, Tianyu Zhang^*, Yuan Gao, Jarek M. van Dijk, Qiuhui Yang, Tao Tan, Ritse M. Mann

^*Corresponding author for this work

GROW - Innovative Cancer Diagnostics & Therapy

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

Abstract

Breast magnetic resonance imaging (MRI) is widely recognized for its high sensitivity in detecting breast cancer. However, interpreting breast MRI scans remains a complex, time-consuming, and resource-intensive task, even for experienced radiologists. To address these challenges, artificial intelligence-based methods are increasingly being employed. In this study, we developed a multimodal breast MRI language-image pre-training (MLIP) approach as an initial exploration of a breast MRI foundation model to aid in the interpretation of scans. Two types of inferences were used to evaluate MLIP’s performance. First, MLIP could retrieve corresponding MRI cases from a dataset based on a query, achieving an area under the receiver operating characteristic curve of 0.717 for suspicious and malignant cases, 0.640 for dense breasts, and 0.601 for low background parenchymal enhancement (BPE). Second, MLIP demonstrated the ability to predict the level of disease suspicion for a given MRI case. The results suggest that MLIP has the potential to serve as a foundation model for breast MRI interpretation. Future work will focus on expanding its capabilities through various downstream tasks and integrating additional models to enhance overall performance.

Original language	English
Title of host publication	Artificial Intelligence and Imaging for Diagnostic and Treatment Challenges in Breast Care - 1st Deep Breast Workshop, Deep-Breath 2024, Held in Conjunction with MICCAI 2024, Proceedings
Editors	Ritse M. Mann, Tianyu Zhang, Luyi Han, Geert Litjens, Tao Tan, Danial Truhn, Shuo Li, Yuan Gao, Shannon Doyle, Robert Martí Marly, Jakob Nikolas Kather, Katja Pinker-Domenig, Shandong Wu
Publisher	Springer Verlag
Pages	42-53
Number of pages	12
Volume	15451 LNCS
ISBN (Print)	9783031777882
DOIs	https://doi.org/10.1007/978-3-031-77789-9_5
Publication status	Published - 1 Jan 2025
Event	1st Deep Breast Workshop on AI and Imaging for Diagnostic and Treatment Challenges in Breast Care, Deep-Breath 2024 - Marrakesh, Morocco Duration: 10 Oct 2024 → 10 Oct 2024 https://deep-breath-miccai.github.io/

Publication series

Series	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	15451 LNCS
ISSN	0302-9743

Workshop

Workshop	1st Deep Breast Workshop on AI and Imaging for Diagnostic and Treatment Challenges in Breast Care, Deep-Breath 2024
Country/Territory	Morocco
City	Marrakesh
Period	10/10/24 → 10/10/24
Internet address	https://deep-breath-miccai.github.io/

Keywords

Breast Imaging
Contrastive Learning
Foundation model
MRI
NLP

Access to Document

10.1007/978-3-031-77789-9_5

Cite this

Rasoolzadeh, N., Zhang, T., Gao, Y., van Dijk, J. M., Yang, Q., Tan, T., & Mann, R. M. (2025). Multimodal Breast MRI Language-Image Pretraining (MLIP): An Exploration of a Breast MRI Foundation Model. In R. M. Mann, T. Zhang, L. Han, G. Litjens, T. Tan, D. Truhn, S. Li, Y. Gao, S. Doyle, R. Martí Marly, J. N. Kather, K. Pinker-Domenig, & S. Wu (Eds.), Artificial Intelligence and Imaging for Diagnostic and Treatment Challenges in Breast Care - 1st Deep Breast Workshop, Deep-Breath 2024, Held in Conjunction with MICCAI 2024, Proceedings (Vol. 15451 LNCS, pp. 42-53). Springer Verlag. https://doi.org/10.1007/978-3-031-77789-9_5

Rasoolzadeh, Nika ; Zhang, Tianyu ; Gao, Yuan et al. / Multimodal Breast MRI Language-Image Pretraining (MLIP) : An Exploration of a Breast MRI Foundation Model. Artificial Intelligence and Imaging for Diagnostic and Treatment Challenges in Breast Care - 1st Deep Breast Workshop, Deep-Breath 2024, Held in Conjunction with MICCAI 2024, Proceedings. editor / Ritse M. Mann ; Tianyu Zhang ; Luyi Han ; Geert Litjens ; Tao Tan ; Danial Truhn ; Shuo Li ; Yuan Gao ; Shannon Doyle ; Robert Martí Marly ; Jakob Nikolas Kather ; Katja Pinker-Domenig ; Shandong Wu. Vol. 15451 LNCS Springer Verlag, 2025. pp. 42-53 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 15451 LNCS).

@inproceedings{fb31d8e3750b4bd7b6d0fff18054c111,

title = "Multimodal Breast MRI Language-Image Pretraining (MLIP): An Exploration of a Breast MRI Foundation Model",

abstract = "Breast magnetic resonance imaging (MRI) is widely recognized for its high sensitivity in detecting breast cancer. However, interpreting breast MRI scans remains a complex, time-consuming, and resource-intensive task, even for experienced radiologists. To address these challenges, artificial intelligence-based methods are increasingly being employed. In this study, we developed a multimodal breast MRI language-image pre-training (MLIP) approach as an initial exploration of a breast MRI foundation model to aid in the interpretation of scans. Two types of inferences were used to evaluate MLIP{\textquoteright}s performance. First, MLIP could retrieve corresponding MRI cases from a dataset based on a query, achieving an area under the receiver operating characteristic curve of 0.717 for suspicious and malignant cases, 0.640 for dense breasts, and 0.601 for low background parenchymal enhancement (BPE). Second, MLIP demonstrated the ability to predict the level of disease suspicion for a given MRI case. The results suggest that MLIP has the potential to serve as a foundation model for breast MRI interpretation. Future work will focus on expanding its capabilities through various downstream tasks and integrating additional models to enhance overall performance.",

keywords = "Breast Imaging, Contrastive Learning, Foundation model, MRI, NLP",

author = "Nika Rasoolzadeh and Tianyu Zhang and Yuan Gao and {van Dijk}, {Jarek M.} and Qiuhui Yang and Tao Tan and Mann, {Ritse M.}",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.; 1st Deep Breast Workshop on AI and Imaging for Diagnostic and Treatment Challenges in Breast Care, Deep-Breath 2024 ; Conference date: 10-10-2024 Through 10-10-2024",

year = "2025",

month = jan,

day = "1",

doi = "10.1007/978-3-031-77789-9_5",

language = "English",

isbn = "9783031777882",

volume = "15451 LNCS",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "42--53",

editor = "Mann, {Ritse M.} and Tianyu Zhang and Luyi Han and Geert Litjens and Tao Tan and Danial Truhn and Shuo Li and Yuan Gao and Shannon Doyle and {Mart{\'i} Marly}, Robert and Kather, {Jakob Nikolas} and Katja Pinker-Domenig and Shandong Wu",

booktitle = "Artificial Intelligence and Imaging for Diagnostic and Treatment Challenges in Breast Care - 1st Deep Breast Workshop, Deep-Breath 2024, Held in Conjunction with MICCAI 2024, Proceedings",

address = "Germany",

url = "https://deep-breath-miccai.github.io/",

}

Rasoolzadeh, N, Zhang, T, Gao, Y, van Dijk, JM, Yang, Q, Tan, T & Mann, RM 2025, Multimodal Breast MRI Language-Image Pretraining (MLIP): An Exploration of a Breast MRI Foundation Model. in RM Mann, T Zhang, L Han, G Litjens, T Tan, D Truhn, S Li, Y Gao, S Doyle, R Martí Marly, JN Kather, K Pinker-Domenig & S Wu (eds), Artificial Intelligence and Imaging for Diagnostic and Treatment Challenges in Breast Care - 1st Deep Breast Workshop, Deep-Breath 2024, Held in Conjunction with MICCAI 2024, Proceedings. vol. 15451 LNCS, Springer Verlag, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 15451 LNCS, pp. 42-53, 1st Deep Breast Workshop on AI and Imaging for Diagnostic and Treatment Challenges in Breast Care, Deep-Breath 2024, Marrakesh, Morocco, 10/10/24. https://doi.org/10.1007/978-3-031-77789-9_5

Multimodal Breast MRI Language-Image Pretraining (MLIP): An Exploration of a Breast MRI Foundation Model. / Rasoolzadeh, Nika; Zhang, Tianyu; Gao, Yuan et al.
Artificial Intelligence and Imaging for Diagnostic and Treatment Challenges in Breast Care - 1st Deep Breast Workshop, Deep-Breath 2024, Held in Conjunction with MICCAI 2024, Proceedings. ed. / Ritse M. Mann; Tianyu Zhang; Luyi Han; Geert Litjens; Tao Tan; Danial Truhn; Shuo Li; Yuan Gao; Shannon Doyle; Robert Martí Marly; Jakob Nikolas Kather; Katja Pinker-Domenig; Shandong Wu. Vol. 15451 LNCS Springer Verlag, 2025. p. 42-53 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 15451 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

TY - GEN

T1 - Multimodal Breast MRI Language-Image Pretraining (MLIP)

T2 - 1st Deep Breast Workshop on AI and Imaging for Diagnostic and Treatment Challenges in Breast Care, Deep-Breath 2024

AU - Rasoolzadeh, Nika

AU - Zhang, Tianyu

AU - Gao, Yuan

AU - van Dijk, Jarek M.

AU - Yang, Qiuhui

AU - Tan, Tao

AU - Mann, Ritse M.

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

PY - 2025/1/1

Y1 - 2025/1/1

N2 - Breast magnetic resonance imaging (MRI) is widely recognized for its high sensitivity in detecting breast cancer. However, interpreting breast MRI scans remains a complex, time-consuming, and resource-intensive task, even for experienced radiologists. To address these challenges, artificial intelligence-based methods are increasingly being employed. In this study, we developed a multimodal breast MRI language-image pre-training (MLIP) approach as an initial exploration of a breast MRI foundation model to aid in the interpretation of scans. Two types of inferences were used to evaluate MLIP’s performance. First, MLIP could retrieve corresponding MRI cases from a dataset based on a query, achieving an area under the receiver operating characteristic curve of 0.717 for suspicious and malignant cases, 0.640 for dense breasts, and 0.601 for low background parenchymal enhancement (BPE). Second, MLIP demonstrated the ability to predict the level of disease suspicion for a given MRI case. The results suggest that MLIP has the potential to serve as a foundation model for breast MRI interpretation. Future work will focus on expanding its capabilities through various downstream tasks and integrating additional models to enhance overall performance.

AB - Breast magnetic resonance imaging (MRI) is widely recognized for its high sensitivity in detecting breast cancer. However, interpreting breast MRI scans remains a complex, time-consuming, and resource-intensive task, even for experienced radiologists. To address these challenges, artificial intelligence-based methods are increasingly being employed. In this study, we developed a multimodal breast MRI language-image pre-training (MLIP) approach as an initial exploration of a breast MRI foundation model to aid in the interpretation of scans. Two types of inferences were used to evaluate MLIP’s performance. First, MLIP could retrieve corresponding MRI cases from a dataset based on a query, achieving an area under the receiver operating characteristic curve of 0.717 for suspicious and malignant cases, 0.640 for dense breasts, and 0.601 for low background parenchymal enhancement (BPE). Second, MLIP demonstrated the ability to predict the level of disease suspicion for a given MRI case. The results suggest that MLIP has the potential to serve as a foundation model for breast MRI interpretation. Future work will focus on expanding its capabilities through various downstream tasks and integrating additional models to enhance overall performance.

KW - Breast Imaging

KW - Contrastive Learning

KW - Foundation model

KW - MRI

KW - NLP

U2 - 10.1007/978-3-031-77789-9_5

DO - 10.1007/978-3-031-77789-9_5

M3 - Conference article in proceeding

SN - 9783031777882

VL - 15451 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 42

EP - 53

BT - Artificial Intelligence and Imaging for Diagnostic and Treatment Challenges in Breast Care - 1st Deep Breast Workshop, Deep-Breath 2024, Held in Conjunction with MICCAI 2024, Proceedings

A2 - Mann, Ritse M.

A2 - Zhang, Tianyu

A2 - Han, Luyi

A2 - Litjens, Geert

A2 - Tan, Tao

A2 - Truhn, Danial

A2 - Li, Shuo

A2 - Gao, Yuan

A2 - Doyle, Shannon

A2 - Martí Marly, Robert

A2 - Kather, Jakob Nikolas

A2 - Pinker-Domenig, Katja

A2 - Wu, Shandong

PB - Springer Verlag

Y2 - 10 October 2024 through 10 October 2024

ER -

Rasoolzadeh N, Zhang T, Gao Y, van Dijk JM, Yang Q, Tan T et al. Multimodal Breast MRI Language-Image Pretraining (MLIP): An Exploration of a Breast MRI Foundation Model. In Mann RM, Zhang T, Han L, Litjens G, Tan T, Truhn D, Li S, Gao Y, Doyle S, Martí Marly R, Kather JN, Pinker-Domenig K, Wu S, editors, Artificial Intelligence and Imaging for Diagnostic and Treatment Challenges in Breast Care - 1st Deep Breast Workshop, Deep-Breath 2024, Held in Conjunction with MICCAI 2024, Proceedings. Vol. 15451 LNCS. Springer Verlag. 2025. p. 42-53. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 15451 LNCS). doi: 10.1007/978-3-031-77789-9_5

Multimodal Breast MRI Language-Image Pretraining (MLIP): An Exploration of a Breast MRI Foundation Model

Abstract

Publication series

Workshop

Keywords

Access to Document

Fingerprint

Cite this