-
Repurposing the scientific literature with vision-language models
Authors:
Anton Alyakin,
Jaden Stryker,
Daniel Alexander Alber,
Karl L. Sangwon,
Jin Vivian Lee,
Brandon Duderstadt,
Akshay Save,
David Kurland,
Spencer Frome,
Shrutika Singh,
Jeff Zhang,
Eunice Yang,
Ki Yun Park,
Cordelia Orillac,
Aly A. Valliani,
Sean Neifert,
Albert Liu,
Aneek Patel,
Christopher Livia,
Darryl Lau,
Ilya Laufer,
Peter A. Rozman,
Eveline Teresa Hidalgo,
Howard Riina,
Rui Feng
, et al. (7 additional authors not shown)
Abstract:
Leading vision-language models (VLMs) are trained on general Internet content, overlooking scientific journals' rich, domain-specific knowledge. Training on specialty-specific literature could yield high-performance, task-specific tools, enabling generative AI to match generalist models in specialty publishing, educational, and clinical tasks. We created NeuroPubs, a multimodal dataset of 23,000 N…
▽ More
Leading vision-language models (VLMs) are trained on general Internet content, overlooking scientific journals' rich, domain-specific knowledge. Training on specialty-specific literature could yield high-performance, task-specific tools, enabling generative AI to match generalist models in specialty publishing, educational, and clinical tasks. We created NeuroPubs, a multimodal dataset of 23,000 Neurosurgery Publications articles (134M words, 78K image-caption pairs). Using NeuroPubs, VLMs generated publication-ready graphical abstracts (70% of 100 abstracts) and board-style questions indistinguishable from human-written ones (54% of 89,587 questions). We used these questions to train CNS-Obsidian, a 34B-parameter VLM. In a blinded, randomized controlled trial, our model demonstrated non-inferiority to then state-of-the-art GPT-4o in neurosurgical differential diagnosis (clinical utility, 40.62% upvotes vs. 57.89%, p=0.1150; accuracy, 59.38% vs. 65.79%, p=0.3797). Our pilot study demonstrates how training generative AI models on specialty-specific journal content - without large-scale internet data - results in high-performance academic and clinical tools, enabling domain-tailored AI across diverse fields.
△ Less
Submitted 27 April, 2025; v1 submitted 26 February, 2025;
originally announced February 2025.
-
Artificial-intelligence-based molecular classification of diffuse gliomas using rapid, label-free optical imaging
Authors:
Todd C. Hollon,
Cheng Jiang,
Asadur Chowdury,
Mustafa Nasir-Moin,
Akhil Kondepudi,
Alexander Aabedi,
Arjun Adapa,
Wajd Al-Holou,
Jason Heth,
Oren Sagher,
Pedro Lowenstein,
Maria Castro,
Lisa Irina Wadiura,
Georg Widhalm,
Volker Neuschmelting,
David Reinecke,
Niklas von Spreckelsen,
Mitchel S. Berger,
Shawn L. Hervey-Jumper,
John G. Golfinos,
Matija Snuderl,
Sandra Camelo-Piragua,
Christian Freudiger,
Honglak Lee,
Daniel A. Orringer
Abstract:
Molecular classification has transformed the management of brain tumors by enabling more accurate prognostication and personalized treatment. However, timely molecular diagnostic testing for patients with brain tumors is limited, complicating surgical and adjuvant treatment and obstructing clinical trial enrollment. In this study, we developed DeepGlioma, a rapid ($< 90$ seconds), artificial-intel…
▽ More
Molecular classification has transformed the management of brain tumors by enabling more accurate prognostication and personalized treatment. However, timely molecular diagnostic testing for patients with brain tumors is limited, complicating surgical and adjuvant treatment and obstructing clinical trial enrollment. In this study, we developed DeepGlioma, a rapid ($< 90$ seconds), artificial-intelligence-based diagnostic screening system to streamline the molecular diagnosis of diffuse gliomas. DeepGlioma is trained using a multimodal dataset that includes stimulated Raman histology (SRH); a rapid, label-free, non-consumptive, optical imaging method; and large-scale, public genomic data. In a prospective, multicenter, international testing cohort of patients with diffuse glioma ($n=153$) who underwent real-time SRH imaging, we demonstrate that DeepGlioma can predict the molecular alterations used by the World Health Organization to define the adult-type diffuse glioma taxonomy (IDH mutation, 1p19q co-deletion and ATRX mutation), achieving a mean molecular classification accuracy of $93.3\pm 1.6\%$. Our results represent how artificial intelligence and optical histology can be used to provide a rapid and scalable adjunct to wet lab methods for the molecular screening of patients with diffuse glioma.
△ Less
Submitted 23 March, 2023;
originally announced March 2023.
-
Rapid Automated Analysis of Skull Base Tumor Specimens Using Intraoperative Optical Imaging and Artificial Intelligence
Authors:
Cheng Jiang,
Abhishek Bhattacharya,
Joseph Linzey,
Rushikesh S. Joshi,
Sung Jik Cha,
Sudharsan Srinivasan,
Daniel Alber,
Akhil Kondepudi,
Esteban Urias,
Balaji Pandian,
Wajd Al-Holou,
Steve Sullivan,
B. Gregory Thompson,
Jason Heth,
Chris Freudiger,
Siri Khalsa,
Donato Pacione,
John G. Golfinos,
Sandra Camelo-Piragua,
Daniel A. Orringer,
Honglak Lee,
Todd Hollon
Abstract:
Background: Accurate diagnosis of skull base tumors is essential for providing personalized surgical treatment strategies. Intraoperative diagnosis can be challenging due to tumor diversity and lack of intraoperative pathology resources.
Objective: To develop an independent and parallel intraoperative pathology workflow that can provide rapid and accurate skull base tumor diagnoses using label-f…
▽ More
Background: Accurate diagnosis of skull base tumors is essential for providing personalized surgical treatment strategies. Intraoperative diagnosis can be challenging due to tumor diversity and lack of intraoperative pathology resources.
Objective: To develop an independent and parallel intraoperative pathology workflow that can provide rapid and accurate skull base tumor diagnoses using label-free optical imaging and artificial intelligence.
Method: We used a fiber laser-based, label-free, non-consumptive, high-resolution microscopy method ($<$ 60 sec per 1 $\times$ 1 mm$^\text{2}$), called stimulated Raman histology (SRH), to image a consecutive, multicenter cohort of skull base tumor patients. SRH images were then used to train a convolutional neural network (CNN) model using three representation learning strategies: cross-entropy, self-supervised contrastive learning, and supervised contrastive learning. Our trained CNN models were tested on a held-out, multicenter SRH dataset.
Results: SRH was able to image the diagnostic features of both benign and malignant skull base tumors. Of the three representation learning strategies, supervised contrastive learning most effectively learned the distinctive and diagnostic SRH image features for each of the skull base tumor types. In our multicenter testing set, cross-entropy achieved an overall diagnostic accuracy of 91.5%, self-supervised contrastive learning 83.9%, and supervised contrastive learning 96.6%. Our trained model was able to identify tumor-normal margins and detect regions of microscopic tumor infiltration in whole-slide SRH images.
Conclusion: SRH with trained artificial intelligence models can provide rapid and accurate intraoperative analysis of skull base tumor specimens to inform surgical decision-making.
△ Less
Submitted 19 June, 2022; v1 submitted 7 August, 2021;
originally announced August 2021.
