MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imaging
Authors:
Noel C. F. Codella,
Ying Jin,
Shrey Jain,
Yu Gu,
Ho Hin Lee,
Asma Ben Abacha,
Alberto Santamaria-Pang,
Will Guyman,
Naiteek Sangani,
Sheng Zhang,
Hoifung Poon,
Stephanie Hyland,
Shruthi Bannur,
Javier Alvarez-Valle,
Xue Li,
John Garrett,
Alan McMillan,
Gaurav Rajguru,
Madhu Maddi,
Nilesh Vijayrania,
Rehaan Bhimai,
Nick Mecklenburg,
Rupal Jain,
Daniel Holstein,
Naveen Gaur
, et al. (6 additional authors not shown)
Abstract:
In this work, we present MedImageInsight, an open-source medical imaging embedding model. MedImageInsight is trained on medical images with associated text and labels across a diverse collection of domains, including X-Ray, CT, MRI, dermoscopy, OCT, fundus photography, ultrasound, histopathology, and mammography. Rigorous evaluations demonstrate MedImageInsight's ability to achieve state-of-the-ar…
▽ More
In this work, we present MedImageInsight, an open-source medical imaging embedding model. MedImageInsight is trained on medical images with associated text and labels across a diverse collection of domains, including X-Ray, CT, MRI, dermoscopy, OCT, fundus photography, ultrasound, histopathology, and mammography. Rigorous evaluations demonstrate MedImageInsight's ability to achieve state-of-the-art (SOTA) or human expert level performance across classification, image-image search, and fine-tuning tasks. Specifically, on public datasets, MedImageInsight achieves SOTA in CT 3D medical image retrieval, as well as SOTA in disease classification and search for chest X-ray, dermatology, and OCT imaging. Furthermore, MedImageInsight achieves human expert performance in bone age estimation (on both public and partner data), as well as AUC above 0.9 in most other domains. When paired with a text decoder, MedImageInsight achieves near SOTA level single image report findings generation with less than 10\% the parameters of other models. Compared to fine-tuning GPT-4o with only MIMIC-CXR data for the same task, MedImageInsight outperforms in clinical metrics, but underperforms on lexical metrics where GPT-4o sets a new SOTA. Importantly for regulatory purposes, MedImageInsight can generate ROC curves, adjust sensitivity and specificity based on clinical need, and provide evidence-based decision support through image-image search (which can also enable retrieval augmented generation). In an independent clinical evaluation of image-image search in chest X-ray, MedImageInsight outperformed every other publicly available foundation model evaluated by large margins (over 6 points AUC), and significantly outperformed other models in terms of AI fairness (across age and gender). We hope releasing MedImageInsight will help enhance collective progress in medical imaging AI research and development.
△ Less
Submitted 9 October, 2024;
originally announced October 2024.
Hierarchical Analysis of Visual COVID-19 Features from Chest Radiographs
Authors:
Shruthi Bannur,
Ozan Oktay,
Melanie Bernhardt,
Anton Schwaighofer,
Rajesh Jena,
Besmira Nushi,
Sharan Wadhwani,
Aditya Nori,
Kal Natarajan,
Shazad Ashraf,
Javier Alvarez-Valle,
Daniel C. Castro
Abstract:
Chest radiography has been a recommended procedure for patient triaging and resource management in intensive care units (ICUs) throughout the COVID-19 pandemic. The machine learning efforts to augment this workflow have been long challenged due to deficiencies in reporting, model evaluation, and failure mode analysis. To address some of those shortcomings, we model radiological features with a hum…
▽ More
Chest radiography has been a recommended procedure for patient triaging and resource management in intensive care units (ICUs) throughout the COVID-19 pandemic. The machine learning efforts to augment this workflow have been long challenged due to deficiencies in reporting, model evaluation, and failure mode analysis. To address some of those shortcomings, we model radiological features with a human-interpretable class hierarchy that aligns with the radiological decision process. Also, we propose the use of a data-driven error analysis methodology to uncover the blind spots of our model, providing further transparency on its clinical utility. For example, our experiments show that model failures highly correlate with ICU imaging conditions and with the inherent difficulty in distinguishing certain types of radiological features. Also, our hierarchical interpretation and analysis facilitates the comparison with respect to radiologists' findings and inter-variability, which in return helps us to better assess the clinical applicability of models.
△ Less
Submitted 14 July, 2021;
originally announced July 2021.
