-
Multi-Modal Latent Variables for Cross-Individual Primary Visual Cortex Modeling and Analysis
Authors:
Yu Zhu,
Bo Lei,
Chunfeng Song,
Wanli Ouyang,
Shan Yu,
Tiejun Huang
Abstract:
Elucidating the functional mechanisms of the primary visual cortex (V1) remains a fundamental challenge in systems neuroscience. Current computational models face two critical limitations, namely the challenge of cross-modal integration between partial neural recordings and complex visual stimuli, and the inherent variability in neural characteristics across individuals, including differences in n…
▽ More
Elucidating the functional mechanisms of the primary visual cortex (V1) remains a fundamental challenge in systems neuroscience. Current computational models face two critical limitations, namely the challenge of cross-modal integration between partial neural recordings and complex visual stimuli, and the inherent variability in neural characteristics across individuals, including differences in neuron populations and firing patterns. To address these challenges, we present a multi-modal identifiable variational autoencoder (miVAE) that employs a two-level disentanglement strategy to map neural activity and visual stimuli into a unified latent space. This framework enables robust identification of cross-modal correlations through refined latent space modeling. We complement this with a novel score-based attribution analysis that traces latent variables back to their origins in the source data space. Evaluation on a large-scale mouse V1 dataset demonstrates that our method achieves state-of-the-art performance in cross-individual latent representation and alignment, without requiring subject-specific fine-tuning, and exhibits improved performance with increasing data size. Significantly, our attribution algorithm successfully identifies distinct neuronal subpopulations characterized by unique temporal patterns and stimulus discrimination properties, while simultaneously revealing stimulus regions that show specific sensitivity to edge features and luminance variations. This scalable framework offers promising applications not only for advancing V1 research but also for broader investigations in neuroscience.
△ Less
Submitted 19 December, 2024;
originally announced December 2024.
-
CATD: Unified Representation Learning for EEG-to-fMRI Cross-Modal Generation
Authors:
Weiheng Yao,
Zhihan Lyu,
Mufti Mahmud,
Ning Zhong,
Baiying Lei,
Shuqiang Wang
Abstract:
Multi-modal neuroimaging analysis is crucial for a comprehensive understanding of brain function and pathology, as it allows for the integration of different imaging techniques, thus overcoming the limitations of individual modalities. However, the high costs and limited availability of certain modalities pose significant challenges. To address these issues, this paper proposes the Condition-Align…
▽ More
Multi-modal neuroimaging analysis is crucial for a comprehensive understanding of brain function and pathology, as it allows for the integration of different imaging techniques, thus overcoming the limitations of individual modalities. However, the high costs and limited availability of certain modalities pose significant challenges. To address these issues, this paper proposes the Condition-Aligned Temporal Diffusion (CATD) framework for end-to-end cross-modal synthesis of neuroimaging, enabling the generation of functional magnetic resonance imaging (fMRI)-detected Blood Oxygen Level Dependent (BOLD) signals from more accessible Electroencephalography (EEG) signals. By constructing Conditionally Aligned Block (CAB), heterogeneous neuroimages are aligned into a latent space, achieving a unified representation that provides the foundation for cross-modal transformation in neuroimaging. The combination with the constructed Dynamic Time-Frequency Segmentation (DTFS) module also enables the use of EEG signals to improve the temporal resolution of BOLD signals, thus augmenting the capture of the dynamic details of the brain. Experimental validation demonstrates that the framework improves the accuracy of brain activity state prediction by 9.13% (reaching 69.8%), enhances the diagnostic accuracy of brain disorders by 4.10% (reaching 99.55%), effectively identifies abnormal brain regions, enhancing the temporal resolution of BOLD signals. The proposed framework establishes a new paradigm for cross-modal synthesis of neuroimaging by unifying heterogeneous neuroimaging data into a latent representation space, showing promise in medical applications such as improving Parkinson's disease prediction and identifying abnormal brain regions.
△ Less
Submitted 25 March, 2025; v1 submitted 16 July, 2024;
originally announced August 2024.
-
BDHT: Generative AI Enables Causality Analysis for Mild Cognitive Impairment
Authors:
Qiankun Zuo,
Ling Chen,
Yanyan Shen,
Michael Kwok-Po Ng,
Baiying Lei,
Shuqiang Wang
Abstract:
Effective connectivity estimation plays a crucial role in understanding the interactions and information flow between different brain regions. However, the functional time series used for estimating effective connectivity is derived from certain software, which may lead to large computing errors because of different parameter settings and degrade the ability to model complex causal relationships b…
▽ More
Effective connectivity estimation plays a crucial role in understanding the interactions and information flow between different brain regions. However, the functional time series used for estimating effective connectivity is derived from certain software, which may lead to large computing errors because of different parameter settings and degrade the ability to model complex causal relationships between brain regions. In this paper, a brain diffuser with hierarchical transformer (BDHT) is proposed to estimate effective connectivity for mild cognitive impairment (MCI) analysis. To our best knowledge, the proposed brain diffuser is the first generative model to apply diffusion models to the application of generating and analyzing multimodal brain networks. Specifically, the BDHT leverages structural connectivity to guide the reverse processes in an efficient way. It makes the denoising process more reliable and guarantees effective connectivity estimation accuracy. To improve denoising quality, the hierarchical denoising transformer is designed to learn multi-scale features in topological space. By stacking the multi-head attention and graph convolutional network, the graph convolutional transformer (GraphConformer) module is devised to enhance structure-function complementarity and improve the ability in noise estimation. Experimental evaluations of the denoising diffusion model demonstrate its effectiveness in estimating effective connectivity. The proposed model achieves superior performance in terms of accuracy and robustness compared to existing approaches. Moreover, the proposed model can identify altered directional connections and provide a comprehensive understanding of parthenogenesis for MCI treatment.
△ Less
Submitted 28 May, 2024; v1 submitted 14 December, 2023;
originally announced December 2023.
-
Brain Structure-Function Fusing Representation Learning using Adversarial Decomposed-VAE for Analyzing MCI
Authors:
Qiankun Zuo,
Baiying Lei,
Ning Zhong,
Yi Pan,
Shuqiang Wang
Abstract:
Integrating the brain structural and functional connectivity features is of great significance in both exploring brain science and analyzing cognitive impairment clinically. However, it remains a challenge to effectively fuse structural and functional features in exploring the brain network. In this paper, a novel brain structure-function fusing-representation learning (BSFL) model is proposed to…
▽ More
Integrating the brain structural and functional connectivity features is of great significance in both exploring brain science and analyzing cognitive impairment clinically. However, it remains a challenge to effectively fuse structural and functional features in exploring the brain network. In this paper, a novel brain structure-function fusing-representation learning (BSFL) model is proposed to effectively learn fused representation from diffusion tensor imaging (DTI) and resting-state functional magnetic resonance imaging (fMRI) for mild cognitive impairment (MCI) analysis. Specifically, the decomposition-fusion framework is developed to first decompose the feature space into the union of the uniform and the unique spaces for each modality, and then adaptively fuse the decomposed features to learn MCI-related representation. Moreover, a knowledge-aware transformer module is designed to automatically capture local and global connectivity features throughout the brain. Also, a uniform-unique contrastive loss is further devised to make the decomposition more effective and enhance the complementarity of structural and functional features. The extensive experiments demonstrate that the proposed model achieves better performance than other competitive methods in predicting and analyzing MCI. More importantly, the proposed model could be a potential tool for reconstructing unified brain networks and predicting abnormal connections during the degenerative processes in MCI.
△ Less
Submitted 23 May, 2023;
originally announced May 2023.
-
SANet:Superpixel Attention Network for Skin Lesion Attributes Detection
Authors:
Xinzi He,
Baiying Lei,
Tianfu Wang
Abstract:
The accurate detection of lesion attributes is meaningful for both the computeraid diagnosis system and dermatologists decisions. However, unlike lesion segmentation and melenoma classification, there are few deep learning methods and literatures focusing on this task. Currently, the lesion attribute detection still remains challenging due to the extremely unbalanced class distribution and insuffi…
▽ More
The accurate detection of lesion attributes is meaningful for both the computeraid diagnosis system and dermatologists decisions. However, unlike lesion segmentation and melenoma classification, there are few deep learning methods and literatures focusing on this task. Currently, the lesion attribute detection still remains challenging due to the extremely unbalanced class distribution and insufficient samples, as well as large intraclass and low interclass variations. To solve these problems, we propose a deep learning framework named superpixel attention network (SANet). Firstly, we segment input images into small regions and shuffle the obtained regions by the random shuttle mechanism (RSM). Secondly, we apply the SANet to capture discriminative features and reconstruct input images. Specifically, SANet contains two sub modules: superpixel average pooling and superpixel at tention module. We introduce a superpixel average pooling to reformulate the superpixel classification problem as a superpixel segmentation problem and a SAMis utilized to focus on discriminative superpixel regions and feature channels. Finally, we design a novel but effective loss, namely global balancing loss to address the serious data imbalance in ISIC 2018 Task 2 lesion attributes detection dataset. The proposed method achieves quite good performance on the ISIC 2018 Task 2 challenge.
△ Less
Submitted 20 October, 2019;
originally announced October 2019.
