-
Evaluating Vision Language Models (VLMs) for Radiology: A Comprehensive Analysis
Authors:
Frank Li,
Hari Trivedi,
Bardia Khosravi,
Theo Dapamede,
Mohammadreza Chavoshi,
Abdulhameed Dere,
Rohan Satya Isaac,
Aawez Mansuri,
Janice Newsome,
Saptarshi Purkayastha,
Judy Gichoya
Abstract:
Foundation models, trained on vast amounts of data using self-supervised techniques, have emerged as a promising frontier for advancing artificial intelligence (AI) applications in medicine. This study evaluates three different vision-language foundation models (RAD-DINO, CheXagent, and BiomedCLIP) on their ability to capture fine-grained imaging features for radiology tasks. The models were asses…
▽ More
Foundation models, trained on vast amounts of data using self-supervised techniques, have emerged as a promising frontier for advancing artificial intelligence (AI) applications in medicine. This study evaluates three different vision-language foundation models (RAD-DINO, CheXagent, and BiomedCLIP) on their ability to capture fine-grained imaging features for radiology tasks. The models were assessed across classification, segmentation, and regression tasks for pneumothorax and cardiomegaly on chest radiographs. Self-supervised RAD-DINO consistently excelled in segmentation tasks, while text-supervised CheXagent demonstrated superior classification performance. BiomedCLIP showed inconsistent performance across tasks. A custom segmentation model that integrates global and local features substantially improved performance for all foundation models, particularly for challenging pneumothorax segmentation. The findings highlight that pre-training methodology significantly influences model performance on specific downstream tasks. For fine-grained segmentation tasks, models trained without text supervision performed better, while text-supervised models offered advantages in classification and interpretability. These insights provide guidance for selecting foundation models based on specific clinical applications in radiology.
△ Less
Submitted 22 April, 2025;
originally announced April 2025.
-
Novel AI-Based Quantification of Breast Arterial Calcification to Predict Cardiovascular Risk
Authors:
Theodorus Dapamede,
Aisha Urooj,
Vedant Joshi,
Gabrielle Gershon,
Frank Li,
Mohammadreza Chavoshi,
Beatrice Brown-Mulry,
Rohan Satya Isaac,
Aawez Mansuri,
Chad Robichaux,
Chadi Ayoub,
Reza Arsanjani,
Laurence Sperling,
Judy Gichoya,
Marly van Assen,
Charles W. ONeill,
Imon Banerjee,
Hari Trivedi
Abstract:
Women are underdiagnosed and undertreated for cardiovascular disease. Automatic quantification of breast arterial calcification on screening mammography can identify women at risk for cardiovascular disease and enable earlier treatment and management of disease. In this retrospective study of 116,135 women from two healthcare systems, a transformer-based neural network quantified BAC severity (no…
▽ More
Women are underdiagnosed and undertreated for cardiovascular disease. Automatic quantification of breast arterial calcification on screening mammography can identify women at risk for cardiovascular disease and enable earlier treatment and management of disease. In this retrospective study of 116,135 women from two healthcare systems, a transformer-based neural network quantified BAC severity (no BAC, mild, moderate, and severe) on screening mammograms. Outcomes included major adverse cardiovascular events (MACE) and all-cause mortality. BAC severity was independently associated with MACE after adjusting for cardiovascular risk factors, with increasing hazard ratios from mild (HR 1.18-1.22), moderate (HR 1.38-1.47), to severe BAC (HR 2.03-2.22) across datasets (all p<0.001). This association remained significant across all age groups, with even mild BAC indicating increased risk in women under 50. BAC remained an independent predictor when analyzed alongside ASCVD risk scores, showing significant associations with myocardial infarction, stroke, heart failure, and mortality (all p<0.005). Automated BAC quantification enables opportunistic cardiovascular risk assessment during routine mammography without additional radiation or cost. This approach provides value beyond traditional risk factors, particularly in younger women, offering potential for early CVD risk stratification in the millions of women undergoing annual mammography.
△ Less
Submitted 17 March, 2025;
originally announced March 2025.
-
Subgroup Performance of a Commercial Digital Breast Tomosynthesis Model for Breast Cancer Detection
Authors:
Beatrice Brown-Mulry,
Rohan Satya Isaac,
Sang Hyup Lee,
Ambika Seth,
KyungJee Min,
Theo Dapamede,
Frank Li,
Aawez Mansuri,
MinJae Woo,
Christian Allison Fauria-Robinson,
Bhavna Paryani,
Judy Wawira Gichoya,
Hari Trivedi
Abstract:
While research has established the potential of AI models for mammography to improve breast cancer screening outcomes, there have not been any detailed subgroup evaluations performed to assess the strengths and weaknesses of commercial models for digital breast tomosynthesis (DBT) imaging. This study presents a granular evaluation of the Lunit INSIGHT DBT model on a large retrospective cohort of 1…
▽ More
While research has established the potential of AI models for mammography to improve breast cancer screening outcomes, there have not been any detailed subgroup evaluations performed to assess the strengths and weaknesses of commercial models for digital breast tomosynthesis (DBT) imaging. This study presents a granular evaluation of the Lunit INSIGHT DBT model on a large retrospective cohort of 163,449 screening mammography exams from the Emory Breast Imaging Dataset (EMBED). Model performance was evaluated in a binary context with various negative exam types (162,081 exams) compared against screen detected cancers (1,368 exams) as the positive class. The analysis was stratified across demographic, imaging, and pathologic subgroups to identify potential disparities. The model achieved an overall AUC of 0.91 (95% CI: 0.90-0.92) with a precision of 0.08 (95% CI: 0.08-0.08), and a recall of 0.73 (95% CI: 0.71-0.76). Performance was found to be robust across demographics, but cases with non-invasive cancers (AUC: 0.85, 95% CI: 0.83-0.87), calcifications (AUC: 0.80, 95% CI: 0.78-0.82), and dense breast tissue (AUC: 0.90, 95% CI: 0.88-0.91) were associated with significantly lower performance compared to other groups. These results highlight the need for detailed evaluation of model characteristics and vigilance in considering adoption of new tools for clinical deployment.
△ Less
Submitted 17 March, 2025;
originally announced March 2025.
-
Emory Knee Radiograph (MRKR) Dataset
Authors:
Brandon Price,
Jason Adleberg,
Kaesha Thomas,
Zach Zaiman,
Aawez Mansuri,
Beatrice Brown-Mulry,
Chima Okecheukwu,
Judy Gichoya,
Hari Trivedi
Abstract:
The Emory Knee Radiograph (MRKR) dataset is a large, demographically diverse collection of 503,261 knee radiographs from 83,011 patients, 40% of which are African American. This dataset provides imaging data in DICOM format along with detailed clinical information, including patient-reported pain scores, diagnostic codes, and procedural codes, which are not commonly available in similar datasets.…
▽ More
The Emory Knee Radiograph (MRKR) dataset is a large, demographically diverse collection of 503,261 knee radiographs from 83,011 patients, 40% of which are African American. This dataset provides imaging data in DICOM format along with detailed clinical information, including patient-reported pain scores, diagnostic codes, and procedural codes, which are not commonly available in similar datasets. The MRKR dataset also features imaging metadata such as image laterality, view type, and presence of hardware, enhancing its value for research and model development. MRKR addresses significant gaps in existing datasets by offering a more representative sample for studying osteoarthritis and related outcomes, particularly among minority populations, thereby providing a valuable resource for clinicians and researchers.
△ Less
Submitted 30 October, 2024;
originally announced November 2024.
-
From Struggle to Simplicity with a Usable and Secure API for Encryption in Java
Authors:
Ehsan Firouzi,
Ammar Mansuri,
Mohammad Ghafari,
Maziar Kaveh
Abstract:
Cryptography misuses are prevalent in the wild. Crypto APIs are hard to use for developers, and static analysis tools do not detect every misuse. We developed SafEncrypt, an API that streamlines encryption tasks for Java developers. It is built on top of the native Java Cryptography Architecture, and it shields developers from crypto complexities and erroneous low-level details. Experiments showed…
▽ More
Cryptography misuses are prevalent in the wild. Crypto APIs are hard to use for developers, and static analysis tools do not detect every misuse. We developed SafEncrypt, an API that streamlines encryption tasks for Java developers. It is built on top of the native Java Cryptography Architecture, and it shields developers from crypto complexities and erroneous low-level details. Experiments showed that SafEncrypt is suitable for developers with varying levels of experience.
△ Less
Submitted 8 September, 2024;
originally announced September 2024.
