ST-Align: A Multimodal Foundation Model for Image-Gene Alignment in Spatial Transcriptomics
Authors:
Yuxiang Lin,
Ling Luo,
Ying Chen,
Xushi Zhang,
Zihui Wang,
Wenxian Yang,
Mengsha Tong,
Rongshan Yu
Abstract:
Spatial transcriptomics (ST) provides high-resolution pathological images and whole-transcriptomic expression profiles at individual spots across whole-slide scales. This setting makes it an ideal data source to develop multimodal foundation models. Although recent studies attempted to fine-tune visual encoders with trainable gene encoders based on spot-level, the absence of a wider slide perspect…
▽ More
Spatial transcriptomics (ST) provides high-resolution pathological images and whole-transcriptomic expression profiles at individual spots across whole-slide scales. This setting makes it an ideal data source to develop multimodal foundation models. Although recent studies attempted to fine-tune visual encoders with trainable gene encoders based on spot-level, the absence of a wider slide perspective and spatial intrinsic relationships limits their ability to capture ST-specific insights effectively. Here, we introduce ST-Align, the first foundation model designed for ST that deeply aligns image-gene pairs by incorporating spatial context, effectively bridging pathological imaging with genomic features. We design a novel pretraining framework with a three-target alignment strategy for ST-Align, enabling (1) multi-scale alignment across image-gene pairs, capturing both spot- and niche-level contexts for a comprehensive perspective, and (2) cross-level alignment of multimodal insights, connecting localized cellular characteristics and broader tissue architecture. Additionally, ST-Align employs specialized encoders tailored to distinct ST contexts, followed by an Attention-Based Fusion Network (ABFN) for enhanced multimodal fusion, effectively merging domain-shared knowledge with ST-specific insights from both pathological and genomic data. We pre-trained ST-Align on 1.3 million spot-niche pairs and evaluated its performance through two downstream tasks across six datasets, demonstrating superior zero-shot and few-shot capabilities. ST-Align highlights the potential for reducing the cost of ST and providing valuable insights into the distinction of critical compositions within human tissue.
△ Less
Submitted 25 November, 2024;
originally announced November 2024.
A Single Model Explains both Visual and Auditory Precortical Coding
Authors:
Honghao Shan,
Matthew H. Tong,
Garrison W. Cottrell
Abstract:
Precortical neural systems encode information collected by the senses, but the driving principles of the encoding used have remained a subject of debate. We present a model of retinal coding that is based on three constraints: information preservation, minimization of the neural wiring, and response equalization. The resulting novel version of sparse principal components analysis successfully capt…
▽ More
Precortical neural systems encode information collected by the senses, but the driving principles of the encoding used have remained a subject of debate. We present a model of retinal coding that is based on three constraints: information preservation, minimization of the neural wiring, and response equalization. The resulting novel version of sparse principal components analysis successfully captures a number of known characteristics of the retinal coding system, such as center-surround receptive fields, color opponency channels, and spatiotemporal responses that correspond to magnocellular and parvocellular pathways. Furthermore, when trained on auditory data, the same model learns receptive fields well fit by gammatone filters, commonly used to model precortical auditory coding. This suggests that efficient coding may be a unifying principle of precortical encoding across modalities.
△ Less
Submitted 7 April, 2016; v1 submitted 26 February, 2016;
originally announced February 2016.
