Developing Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography
Authors:
Ibrahim Ethem Hamamci,
Sezgin Er,
Chenyu Wang,
Furkan Almas,
Ayse Gulnihan Simsek,
Sevval Nil Esirgun,
Irem Doga,
Omer Faruk Durugol,
Weicheng Dai,
Murong Xu,
Muhammed Furkan Dasdelen,
Bastian Wittmann,
Tamaz Amiranashvili,
Enis Simsar,
Mehmet Simsar,
Emine Bensu Erdemir,
Abdullah Alanbay,
Anjany Sekuboyina,
Berkan Lafci,
Christian Bluethgen,
Kayhan Batmanghelich,
Mehmet Kemal Ozdemir,
Bjoern Menze
Abstract:
While computer vision has achieved tremendous success with multimodal encoding and direct textual interaction with images via chat-based large language models, similar advancements in medical imaging AI, particularly in 3D imaging, have been limited due to the scarcity of comprehensive datasets. To address this critical gap, we introduce CT-RATE, the first dataset that pairs 3D medical images with…
▽ More
While computer vision has achieved tremendous success with multimodal encoding and direct textual interaction with images via chat-based large language models, similar advancements in medical imaging AI, particularly in 3D imaging, have been limited due to the scarcity of comprehensive datasets. To address this critical gap, we introduce CT-RATE, the first dataset that pairs 3D medical images with corresponding textual reports. CT-RATE comprises 25,692 non-contrast 3D chest CT scans from 21,304 unique patients. Through various reconstructions, these scans are expanded to 50,188 volumes, totaling over 14.3 million 2D slices. Each scan is accompanied by its corresponding radiology report. Leveraging CT-RATE, we develop CT-CLIP, a CT-focused contrastive language-image pretraining framework designed for broad applications without the need for task-specific training. We demonstrate how CT-CLIP can be used in two tasks: multi-abnormality detection and case retrieval. Remarkably, in multi-abnormality detection, CT-CLIP outperforms state-of-the-art fully supervised models across all key metrics, effectively eliminating the need for manual annotation. In case retrieval, it efficiently retrieves relevant cases using either image or textual queries, thereby enhancing knowledge dissemination. By combining CT-CLIP's vision encoder with a pretrained large language model, we create CT-CHAT, a vision-language foundational chat model for 3D chest CT volumes. Finetuned on over 2.7 million question-answer pairs derived from the CT-RATE dataset, CT-CHAT surpasses other multimodal AI assistants, underscoring the necessity for specialized methods in 3D medical imaging. Collectively, the open-source release of CT-RATE, CT-CLIP, and CT-CHAT not only addresses critical challenges in 3D medical imaging, but also lays the groundwork for future innovations in medical AI and improved patient care.
△ Less
Submitted 4 April, 2025; v1 submitted 26 March, 2024;
originally announced March 2024.
GenerateCT: Text-Conditional Generation of 3D Chest CT Volumes
Authors:
Ibrahim Ethem Hamamci,
Sezgin Er,
Anjany Sekuboyina,
Enis Simsar,
Alperen Tezcan,
Ayse Gulnihan Simsek,
Sevval Nil Esirgun,
Furkan Almas,
Irem Dogan,
Muhammed Furkan Dasdelen,
Chinmay Prabhakar,
Hadrien Reynaud,
Sarthak Pati,
Christian Bluethgen,
Mehmet Kemal Ozdemir,
Bjoern Menze
Abstract:
GenerateCT, the first approach to generating 3D medical imaging conditioned on free-form medical text prompts, incorporates a text encoder and three key components: a novel causal vision transformer for encoding 3D CT volumes, a text-image transformer for aligning CT and text tokens, and a text-conditional super-resolution diffusion model. Without directly comparable methods in 3D medical imaging,…
▽ More
GenerateCT, the first approach to generating 3D medical imaging conditioned on free-form medical text prompts, incorporates a text encoder and three key components: a novel causal vision transformer for encoding 3D CT volumes, a text-image transformer for aligning CT and text tokens, and a text-conditional super-resolution diffusion model. Without directly comparable methods in 3D medical imaging, we benchmarked GenerateCT against cutting-edge methods, demonstrating its superiority across all key metrics. Importantly, we evaluated GenerateCT's clinical applications in a multi-abnormality classification task. First, we established a baseline by training a multi-abnormality classifier on our real dataset. To further assess the model's generalization to external data and performance with unseen prompts in a zero-shot scenario, we employed an external set to train the classifier, setting an additional benchmark. We conducted two experiments in which we doubled the training datasets by synthesizing an equal number of volumes for each set using GenerateCT. The first experiment demonstrated an 11% improvement in the AP score when training the classifier jointly on real and generated volumes. The second experiment showed a 7% improvement when training on both real and generated volumes based on unseen prompts. Moreover, GenerateCT enables the scaling of synthetic training datasets to arbitrary sizes. As an example, we generated 100,000 3D CTs, fivefold the number in our real set, and trained the classifier exclusively on these synthetic CTs. Impressively, this classifier surpassed the performance of the one trained on all available real data by a margin of 8%. Last, domain experts evaluated the generated volumes, confirming a high degree of alignment with the text prompt. Access our code, model weights, training data, and generated data at https://github.com/ibrahimethemhamamci/GenerateCT
△ Less
Submitted 12 July, 2024; v1 submitted 25 May, 2023;
originally announced May 2023.
