-
MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding
Authors:
Weikang Qiu,
Zheng Huang,
Haoyu Hu,
Aosong Feng,
Yujun Yan,
Rex Ying
Abstract:
Decoding functional magnetic resonance imaging (fMRI) signals into text has been a key challenge in the neuroscience community, with the potential to advance brain-computer interfaces and uncover deeper insights into brain mechanisms. However, existing approaches often struggle with suboptimal predictive performance, limited task variety, and poor generalization across subjects. In response to thi…
▽ More
Decoding functional magnetic resonance imaging (fMRI) signals into text has been a key challenge in the neuroscience community, with the potential to advance brain-computer interfaces and uncover deeper insights into brain mechanisms. However, existing approaches often struggle with suboptimal predictive performance, limited task variety, and poor generalization across subjects. In response to this, we propose MindLLM, a model designed for subject-agnostic and versatile fMRI-to-text decoding. MindLLM consists of an fMRI encoder and an off-the-shelf LLM. The fMRI encoder employs a neuroscience-informed attention mechanism, which is capable of accommodating subjects with varying input shapes and thus achieves high-performance subject-agnostic decoding. Moreover, we introduce Brain Instruction Tuning (BIT), a novel approach that enhances the model's ability to capture diverse semantic representations from fMRI signals, facilitating more versatile decoding. We evaluate MindLLM on comprehensive fMRI-to-text benchmarks. Results demonstrate that our model outperforms the baselines, improving downstream tasks by 12.0%, unseen subject generalization by 24.5%, and novel task adaptation by 25.0%. Furthermore, the attention patterns in MindLLM provide interpretable insights into its decision-making process.
△ Less
Submitted 6 June, 2025; v1 submitted 17 February, 2025;
originally announced February 2025.
-
Learning High-Order Relationships of Brain Regions
Authors:
Weikang Qiu,
Huangrui Chu,
Selena Wang,
Haolan Zuo,
Xiaoxiao Li,
Yize Zhao,
Rex Ying
Abstract:
Discovering reliable and informative relationships among brain regions from functional magnetic resonance imaging (fMRI) signals is essential in phenotypic predictions. Most of the current methods fail to accurately characterize those interactions because they only focus on pairwise connections and overlook the high-order relationships of brain regions. We propose that these high-order relationshi…
▽ More
Discovering reliable and informative relationships among brain regions from functional magnetic resonance imaging (fMRI) signals is essential in phenotypic predictions. Most of the current methods fail to accurately characterize those interactions because they only focus on pairwise connections and overlook the high-order relationships of brain regions. We propose that these high-order relationships should be maximally informative and minimally redundant (MIMR). However, identifying such high-order relationships is challenging and under-explored due to the exponential search space and the absence of a tractable objective. In response to this gap, we propose a novel method named HYBRID which aims to extract MIMR high-order relationships from fMRI data. HYBRID employs a CONSTRUCTOR to identify hyperedge structures, and a WEIGHTER to compute a weight for each hyperedge, which avoids searching in exponential space. HYBRID achieves the MIMR objective through an innovative information bottleneck framework named multi-head drop-bottleneck with theoretical guarantees. Our comprehensive experiments demonstrate the effectiveness of our model. Our model outperforms the state-of-the-art predictive model by an average of 11.2%, regarding the quality of hyperedges measured by CPM, a standard protocol for studying brain connections.
△ Less
Submitted 8 June, 2024; v1 submitted 2 December, 2023;
originally announced December 2023.
-
Reconstruction of gene regulatory network via sparse optimization
Authors:
Jiashu Lou,
Leyi Cui,
Wenxuan Qiu
Abstract:
In this paper, we tested several sparse optimization algorithms based on the public dataset of the DREAM5 Gene Regulatory Network Inference Challenge. And we find that introducing 20% of the regulatory network as a priori known data can provide a basis for parameter selection of inference algorithms, thus improving prediction efficiency and accuracy. In addition to testing common sparse optimizati…
▽ More
In this paper, we tested several sparse optimization algorithms based on the public dataset of the DREAM5 Gene Regulatory Network Inference Challenge. And we find that introducing 20% of the regulatory network as a priori known data can provide a basis for parameter selection of inference algorithms, thus improving prediction efficiency and accuracy. In addition to testing common sparse optimization methods, we also developed voting algorithms by bagging them. Experiments on the DREAM5 dataset show that the sparse optimization-based inference of the moderation relation works well, achieving better results than the official DREAM5 results on three datasets. However, the performance of traditional independent algorithms varies greatly in the face of different datasets, while our voting algorithm achieves the best results on three of the four datasets.
△ Less
Submitted 11 November, 2022;
originally announced November 2022.
-
Dynamic Studies of Scaffold-dependent Mating Pathway in Yeast
Authors:
Danying Shao,
Wen Zheng,
Wenjun Qiu,
Qi Ouyang,
Chao Tang
Abstract:
The mating pathway in \emph{Saccharomyces cerevisiae} is one of the best understood signal transduction pathways in eukaryotes. It transmits the mating signal from plasma membrane into the nucleus through the G-protein coupled receptor and the mitogen-activated protein kinase (MAPK) cascade. According to the current understandings of the mating pathway, we construct a system of ordinary differen…
▽ More
The mating pathway in \emph{Saccharomyces cerevisiae} is one of the best understood signal transduction pathways in eukaryotes. It transmits the mating signal from plasma membrane into the nucleus through the G-protein coupled receptor and the mitogen-activated protein kinase (MAPK) cascade. According to the current understandings of the mating pathway, we construct a system of ordinary differential equations to describe the process. Our model is consistent with a wide range of experiments, indicating that it captures some main characteristics of the signal transduction along the pathway. Investigation with the model reveals that the shuttling of the scaffold protein and the dephosphorylation of kinases involved in the MAPK cascade cooperate to regulate the response upon pheromone induction and to help preserving the fidelity of the mating signaling. We explored factors affecting the dose-response curves of this pathway and found that both negative feedback and concentrations of the proteins involved in the MAPK cascade play crucial role. Contrary to some other MAPK systems where signaling sensitivity is being amplified successively along the cascade, here the mating signal is transmitted through the cascade in an almost linear fashion.
△ Less
Submitted 11 September, 2006;
originally announced September 2006.
