-
Multimodal Whole Slide Foundation Model for Pathology
Authors:
Tong Ding,
Sophia J. Wagner,
Andrew H. Song,
Richard J. Chen,
Ming Y. Lu,
Andrew Zhang,
Anurag J. Vaidya,
Guillaume Jaume,
Muhammad Shaban,
Ahrong Kim,
Drew F. K. Williamson,
Bowen Chen,
Cristina Almagro-Perez,
Paul Doucet,
Sharifa Sahai,
Chengkuan Chen,
Daisuke Komura,
Akihiro Kawabe,
Shumpei Ishikawa,
Georg Gerber,
Tingying Peng,
Long Phi Le,
Faisal Mahmood
Abstract:
The field of computational pathology has been transformed with recent advances in foundation models that encode histopathology region-of-interests (ROIs) into versatile and transferable feature representations via self-supervised learning (SSL). However, translating these advancements to address complex clinical challenges at the patient and slide level remains constrained by limited clinical data…
▽ More
The field of computational pathology has been transformed with recent advances in foundation models that encode histopathology region-of-interests (ROIs) into versatile and transferable feature representations via self-supervised learning (SSL). However, translating these advancements to address complex clinical challenges at the patient and slide level remains constrained by limited clinical data in disease-specific cohorts, especially for rare clinical conditions. We propose TITAN, a multimodal whole slide foundation model pretrained using 335,645 WSIs via visual self-supervised learning and vision-language alignment with corresponding pathology reports and 423,122 synthetic captions generated from a multimodal generative AI copilot for pathology. Without any finetuning or requiring clinical labels, TITAN can extract general-purpose slide representations and generate pathology reports that generalize to resource-limited clinical scenarios such as rare disease retrieval and cancer prognosis. We evaluate TITAN on diverse clinical tasks and find that TITAN outperforms both ROI and slide foundation models across machine learning settings such as linear probing, few-shot and zero-shot classification, rare cancer retrieval and cross-modal retrieval, and pathology report generation.
△ Less
Submitted 29 November, 2024;
originally announced November 2024.
-
Object-Attribute Biclustering for Elimination of Missing Genotypes in Ischemic Stroke Genome-Wide Data
Authors:
Dmitry I. Ignatov,
Gennady V. Khvorykh,
Andrey V. Khrunin,
Stefan Nikolić,
Makhmud Shaban,
Elizaveta A. Petrova,
Evgeniya A. Koltsova,
Fouzi Takelait,
Dmitrii Egurnov
Abstract:
Missing genotypes can affect the efficacy of machine learning approaches to identify the risk genetic variants of common diseases and traits. The problem occurs when genotypic data are collected from different experiments with different DNA microarrays, each being characterised by its pattern of uncalled (missing) genotypes. This can prevent the machine learning classifier from assigning the class…
▽ More
Missing genotypes can affect the efficacy of machine learning approaches to identify the risk genetic variants of common diseases and traits. The problem occurs when genotypic data are collected from different experiments with different DNA microarrays, each being characterised by its pattern of uncalled (missing) genotypes. This can prevent the machine learning classifier from assigning the classes correctly. To tackle this issue, we used well-developed notions of object-attribute biclusters and formal concepts that correspond to dense subrelations in the binary relation $\textit{patients} \times \textit{SNPs}$. The paper contains experimental results on applying a biclustering algorithm to a large real-world dataset collected for studying the genetic bases of ischemic stroke. The algorithm could identify large dense biclusters in the genotypic matrix for further processing, which in return significantly improved the quality of machine learning classifiers. The proposed algorithm was also able to generate biclusters for the whole dataset without size constraints in comparison to the In-Close4 algorithm for generation of formal concepts.
△ Less
Submitted 25 October, 2020; v1 submitted 22 October, 2020;
originally announced October 2020.
-
Context-Aware Convolutional Neural Network for Grading of Colorectal Cancer Histology Images
Authors:
Muhammad Shaban,
Ruqayya Awan,
Muhammad Moazam Fraz,
Ayesha Azam,
David Snead,
Nasir M. Rajpoot
Abstract:
Digital histology images are amenable to the application of convolutional neural network (CNN) for analysis due to the sheer size of pixel data present in them. CNNs are generally used for representation learning from small image patches (e.g. 224x224) extracted from digital histology images due to computational and memory constraints. However, this approach does not incorporate high-resolution co…
▽ More
Digital histology images are amenable to the application of convolutional neural network (CNN) for analysis due to the sheer size of pixel data present in them. CNNs are generally used for representation learning from small image patches (e.g. 224x224) extracted from digital histology images due to computational and memory constraints. However, this approach does not incorporate high-resolution contextual information in histology images. We propose a novel way to incorporate larger context by a context-aware neural network based on images with a dimension of 1,792x1,792 pixels. The proposed framework first encodes the local representation of a histology image into high dimensional features then aggregates the features by considering their spatial organization to make a final prediction. The proposed method is evaluated for colorectal cancer grading and breast cancer classification. A comprehensive analysis of some variants of the proposed method is presented. Our method outperformed the traditional patch-based approaches, problem-specific methods, and existing context-based methods quantitatively by a margin of 3.61%. Code and dataset related information is available at this link: https://tia-lab.github.io/Context-Aware-CNN
△ Less
Submitted 22 July, 2019;
originally announced July 2019.
