Automated AI Pipeline for Interpretable Longitudinal Mammography Analysis
Evaluation of morphologic features in breast tissue and suspicious findings, and estimation of its temporal changes are the concomitant processes in mammogram interpretation for breast cancer screening. Artificial intelligent (AI) tools with the ability to automatically detect temporal changes in breast tissues have the potential to increase the specificity and accuracy of breast cancer detection, which is especially beneficial in lower socioeconomic settings with limited access to well-trained mammogram-specialized radiologists. However, current commercial and research computer-assisted detection techniques mostly focus on interpretation of screening mammography at a single time point and cannot efficiently track temporal changes. Preliminary efforts have been devoted to developing AI models using serial exams to diagnose breast cancer, but these models suffer from a common shortcoming of limited interpretability. These gaps limit the generalizability and clinical utility of existing AI tools for mammogram-based cancer detection. To address the unmet need, we propose to develop an AI-based cancer detection pipeline to automatize longitudinal mammogram analysis by integrating multi-modal screening information and demonstrating interpretable characterization of temporal changes in breast tissue and findings.
Final Report
The overall goal of this project is to develop an interpretable AI pipeline for longitudinal mammography analysis, addressing major limitations of current mammography AI, including reliance on single screening time points, limited explanation of the imaging features driving predictions, and the substantial effort required to create high-quality spatial annotations for model development. The work leverages the large, prospectively collected MERIT breast cancer screening cohort from MD Anderson and combines clinical radiology reports with serial mammographic images to make longitudinal imaging data more accessible for large-scale breast cancer detection and biomarker research.
First, we developed the Text2Annotation approach to generate clinically meaningful spatial annotations without requiring time-intensive manual delineation by radiologists. The method uses large language models (LLM) to extract spatially encoded descriptors of breast findings from routinely available free-text radiology reports and converts these descriptors into spatial priors and masks that identify the relevant regions on mammograms. Clinical relevance is evaluated by comparing generated annotations with physician annotations and computer-aided detection outputs. This strategy directly addresses the imbalance between the abundance of routine mammograms and reports and the limited availability of expert-generated spatial annotations. The team is currently synthesizing full-pipeline validation metrics, where the accuracy of LLM-based information extraction and anatomical landmark identification are proved to be highly robust, and we are preparing the work for publication.
Second, we developed MuSTAF, a clinically motivated multi-task spatiotemporal framework for breast cancer classification from longitudinal mammography. MuSTAF integrates multiple recent screening examinations and bilateral CC/MLO views while jointly learning cancer status, breast density, bilateral symmetry, and tumor laterality. In the MERIT cohort, the model achieved an AUC of 0.84 and outperformed several state-of-the-art longitudinal deep-learning architectures. Ablation studies demonstrated that both longitudinal modeling and clinically relevant auxiliary tasks improved cancer discrimination, while external validation emphasized the importance of temporal alignment between mammographic examinations and cancer diagnosis. Our manuscript on this topic is now under review.
Together, these developments closely align with the technical milestones in report-based annotation, longitudinal feature analysis, and dissemination. They establish complementary components of a scalable and interpretable longitudinal mammography platform that can reduce manual annotation requirements, support more diverse imaging cohorts, and provide clinically meaningful spatial and temporal information alongside AI-based cancer predictions.
Presentation:
Li Y, Castelo A, Dennison J, et al. Spatio-Temporal Attention Fusion Model for Breast Cancer Detection. International Symposium on Biomedical Imaging (IEEE ISBI), Abstract, Houston, TX, 2024.
Publication / Preprint
Li Y, Castelo A, Dennison JB, et al. MuSTAF: Clinically Relevant Multi-task Spatiotemporal Attention Fusion Framework for Breast Cancer Detection with Longitudinal Mammography. medRxiv. 2026; 2026.07.07.26357474.