Showing 1–10 of 10 results for author: van Gerven, M A J

Multimodal Integration of Longitudinal Noninvasive Diagnostics for Survival Prediction in Immunotherapy Using Deep Learning

Authors: Melda Yeghaian, Zuhir Bodalal, Daan van den Broek, John B A G Haanen, Regina G H Beets-Tan, Stefano Trebeschi, Marcel A J van Gerven

Abstract: Purpose: Immunotherapies have revolutionized the landscape of cancer treatments. However, our understanding of response patterns in advanced cancers treated with immunotherapy remains limited. By leveraging routinely collected noninvasive longitudinal and multimodal data with artificial intelligence, we could unlock the potential to transform immunotherapy for cancer patients, paving the way for p… ▽ More Purpose: Immunotherapies have revolutionized the landscape of cancer treatments. However, our understanding of response patterns in advanced cancers treated with immunotherapy remains limited. By leveraging routinely collected noninvasive longitudinal and multimodal data with artificial intelligence, we could unlock the potential to transform immunotherapy for cancer patients, paving the way for personalized treatment approaches. Methods: In this study, we developed a novel artificial neural network architecture, multimodal transformer-based simple temporal attention (MMTSimTA) network, building upon a combination of recent successful developments. We integrated pre- and on-treatment blood measurements, prescribed medications and CT-based volumes of organs from a large pan-cancer cohort of 694 patients treated with immunotherapy to predict mortality at three, six, nine and twelve months. Different variants of our extended MMTSimTA network were implemented and compared to baseline methods incorporating intermediate and late fusion based integration methods. Results: The strongest prognostic performance was demonstrated using a variant of the MMTSimTA model with area under the curves (AUCs) of $0.84 \pm $0.04, $0.83 \pm $0.02, $0.82 \pm $0.02, $0.81 \pm $0.03 for 3-, 6-, 9-, and 12-month survival prediction, respectively. Discussion: Our findings show that integrating noninvasive longitudinal data using our novel architecture yields an improved multimodal prognostic performance, especially in short-term survival prediction. Conclusion: Our study demonstrates that multimodal longitudinal integration of noninvasive data using deep learning may offer a promising approach for personalized prognostication in immunotherapy-treated cancer patients. △ Less

Submitted 7 June, 2025; v1 submitted 27 November, 2024; originally announced November 2024.

Journal ref: Journal of the American Medical Informatics Association, 2025;, ocaf074

The neuroconnectionist research programme

Authors: Adrien Doerig, Rowan Sommers, Katja Seeliger, Blake Richards, Jenann Ismael, Grace Lindsay, Konrad Kording, Talia Konkle, Marcel A. J. Van Gerven, Nikolaus Kriegeskorte, Tim C. Kietzmann

Abstract: Artificial Neural Networks (ANNs) inspired by biology are beginning to be widely used to model behavioral and neural data, an approach we call neuroconnectionism. ANNs have been lauded as the current best models of information processing in the brain, but also criticized for failing to account for basic cognitive functions. We propose that arguing about the successes and failures of a restricted s… ▽ More Artificial Neural Networks (ANNs) inspired by biology are beginning to be widely used to model behavioral and neural data, an approach we call neuroconnectionism. ANNs have been lauded as the current best models of information processing in the brain, but also criticized for failing to account for basic cognitive functions. We propose that arguing about the successes and failures of a restricted set of current ANNs is the wrong approach to assess the promise of neuroconnectionism. Instead, we take inspiration from the philosophy of science, and in particular from Lakatos, who showed that the core of scientific research programmes is often not directly falsifiable, but should be assessed by its capacity to generate novel insights. Following this view, we present neuroconnectionism as a cohesive large-scale research programme centered around ANNs as a computational language for expressing falsifiable theories about brain computation. We describe the core of the programme, the underlying computational framework and its tools for testing specific neuroscientific hypotheses. Taking a longitudinal view, we review past and present neuroconnectionist projects and their responses to challenges, and argue that the research programme is highly progressive, generating new and otherwise unreachable insights into the workings of the brain. △ Less

Submitted 8 September, 2022; originally announced September 2022.

Comments: 23 pages, 4 figures

Overcoming the Weight Transport Problem via Spike-Timing-Dependent Weight Inference

Authors: Nasir Ahmad, Luca Ambrogioni, Marcel A. J. van Gerven

Abstract: We propose a solution to the weight transport problem, which questions the biological plausibility of the backpropagation algorithm. We derive our method based upon a theoretical analysis of the (approximate) dynamics of leaky integrate-and-fire neurons. We show that the use of spike timing alone outcompetes existing biologically plausible methods for synaptic weight inference in spiking neural ne… ▽ More We propose a solution to the weight transport problem, which questions the biological plausibility of the backpropagation algorithm. We derive our method based upon a theoretical analysis of the (approximate) dynamics of leaky integrate-and-fire neurons. We show that the use of spike timing alone outcompetes existing biologically plausible methods for synaptic weight inference in spiking neural network models. Furthermore, our proposed method is more flexible, being applicable to any spiking neuron model, is conservative in how many parameters are required for implementation and can be deployed in an online-fashion with minimal computational overhead. These features, together with its biological plausibility, make it an attractive mechanism underlying weight inference at single synapses. △ Less

Submitted 11 August, 2021; v1 submitted 9 March, 2020; originally announced March 2020.

Comments: 20 pages, 6 figures

Modulation of early visual processing alleviates capacity limits in solving multiple tasks

Authors: Sushrut Thorat, Giacomo Aldegheri, Marcel A. J. van Gerven, Marius V. Peelen

Abstract: In daily life situations, we have to perform multiple tasks given a visual stimulus, which requires task-relevant information to be transmitted through our visual system. When it is not possible to transmit all the possibly relevant information to higher layers, due to a bottleneck, task-based modulation of early visual processing might be necessary. In this work, we report how the effectiveness o… ▽ More In daily life situations, we have to perform multiple tasks given a visual stimulus, which requires task-relevant information to be transmitted through our visual system. When it is not possible to transmit all the possibly relevant information to higher layers, due to a bottleneck, task-based modulation of early visual processing might be necessary. In this work, we report how the effectiveness of modulating the early processing stage of an artificial neural network depends on the information bottleneck faced by the network. The bottleneck is quantified by the number of tasks the network has to perform and the neural capacity of the later stage of the network. The effectiveness is gauged by the performance on multiple object detection tasks, where the network is trained with a recent multi-task optimization scheme. By associating neural modulations with task-based switching of the state of the network and characterizing when such switching is helpful in early processing, our results provide a functional perspective towards understanding why task-based modulation of early neural processes might be observed in the primate visual cortex △ Less

Submitted 23 September, 2019; v1 submitted 29 July, 2019; originally announced July 2019.

Comments: Main paper - 4 pages, 2 figures; Appendix - 2 pages, 2 figures; Published at the 2019 Conference on Cognitive Computational Neuroscience

Modeling the dynamics of human brain activity with recurrent neural networks

Authors: Umut Güçlü, Marcel A. J. van Gerven

Abstract: Encoding models are used for predicting brain activity in response to sensory stimuli with the objective of elucidating how sensory information is represented in the brain. Encoding models typically comprise a nonlinear transformation of stimuli to features (feature model) and a linear transformation of features to responses (response model). While there has been extensive work on developing bette… ▽ More Encoding models are used for predicting brain activity in response to sensory stimuli with the objective of elucidating how sensory information is represented in the brain. Encoding models typically comprise a nonlinear transformation of stimuli to features (feature model) and a linear transformation of features to responses (response model). While there has been extensive work on developing better feature models, the work on developing better response models has been rather limited. Here, we investigate the extent to which recurrent neural network models can use their internal memories for nonlinear processing of arbitrary feature sequences to predict feature-evoked response sequences as measured by functional magnetic resonance imaging. We show that the proposed recurrent neural network models can significantly outperform established response models by accurately estimating long-term dependencies that drive hemodynamic responses. The results open a new window into modeling the dynamics of brain activity in response to sensory stimuli. △ Less

Submitted 9 June, 2016; originally announced June 2016.

Brains on Beats

Authors: Umut Güçlü, Jordy Thielen, Michael Hanke, Marcel A. J. van Gerven

Abstract: We developed task-optimized deep neural networks (DNNs) that achieved state-of-the-art performance in different evaluation scenarios for automatic music tagging. These DNNs were subsequently used to probe the neural representations of music. Representational similarity analysis revealed the existence of a representational gradient across the superior temporal gyrus (STG). Anterior STG was shown to… ▽ More We developed task-optimized deep neural networks (DNNs) that achieved state-of-the-art performance in different evaluation scenarios for automatic music tagging. These DNNs were subsequently used to probe the neural representations of music. Representational similarity analysis revealed the existence of a representational gradient across the superior temporal gyrus (STG). Anterior STG was shown to be more sensitive to low-level stimulus features encoded in shallow DNN layers whereas posterior STG was shown to be more sensitive to high-level stimulus features encoded in deep DNN layers. △ Less

Submitted 8 June, 2016; originally announced June 2016.

Dynamic Decomposition of Spatiotemporal Neural Signals

Authors: Luca Ambrogioni, Marcel A. J. van Gerven, Eric Maris

Abstract: Neural signals are characterized by rich temporal and spatiotemporal dynamics that reflect the organization of cortical networks. Theoretical research has shown how neural networks can operate at different dynamic ranges that correspond to specific types of information processing. Here we present a data analysis framework that uses a linearized model of these dynamic states in order to decompose t… ▽ More Neural signals are characterized by rich temporal and spatiotemporal dynamics that reflect the organization of cortical networks. Theoretical research has shown how neural networks can operate at different dynamic ranges that correspond to specific types of information processing. Here we present a data analysis framework that uses a linearized model of these dynamic states in order to decompose the measured neural signal into a series of components that capture both rhythmic and non-rhythmic neural activity. The method is based on stochastic differential equations and Gaussian process regression. Through computer simulations and analysis of magnetoencephalographic data, we demonstrate the efficacy of the method in identifying meaningful modulations of oscillatory signals corrupted by structured temporal and spatiotemporal noise. These results suggest that the method is particularly suitable for the analysis and interpretation of complex temporal and spatiotemporal neural signals. △ Less

Submitted 9 May, 2016; originally announced May 2016.

Semantic vector space models predict neural responses to complex visual stimuli

Authors: Umut Güçlü, Marcel A. J. van Gerven

Abstract: Encoding models have as their objective to predict neural responses to naturalistic stimuli with the aim of elucidating how sensory information is represented in the brain. This prediction is achieved by representing the stimulus in terms of a suitable feature space and using this feature space to linearly predict observed neural responses. Here, we investigate to what extent semantic vector space… ▽ More Encoding models have as their objective to predict neural responses to naturalistic stimuli with the aim of elucidating how sensory information is represented in the brain. This prediction is achieved by representing the stimulus in terms of a suitable feature space and using this feature space to linearly predict observed neural responses. Here, we investigate to what extent semantic vector space models can be used to predict neural responses to complex visual stimuli. We show that these models provide good predictions of neural responses in downstream visual areas, improving significantly over a low-level control model based on Gabor wavelet pyramids. The outlined approach provides a new way to model and map high-level semantic representations across cortex. △ Less

Submitted 15 October, 2015; originally announced October 2015.

Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway

Authors: Umut Güçlü, Marcel A. J. van Gerven

Abstract: Converging evidence suggests that the mammalian ventral visual pathway encodes increasingly complex stimulus features in downstream areas. Using deep convolutional neural networks, we can now quantitatively demonstrate that there is indeed an explicit gradient for feature complexity in the ventral pathway of the human brain. Our approach also allows stimulus features of increasing complexity to be… ▽ More Converging evidence suggests that the mammalian ventral visual pathway encodes increasingly complex stimulus features in downstream areas. Using deep convolutional neural networks, we can now quantitatively demonstrate that there is indeed an explicit gradient for feature complexity in the ventral pathway of the human brain. Our approach also allows stimulus features of increasing complexity to be mapped across the human brain, providing an automated approach to probing how representations are mapped across the cortical sheet. Finally, it is shown that deep convolutional neural networks allow decoding of representations in the human brain at a previously unattainable degree of accuracy, providing a more sensitive window into the human brain. △ Less

Submitted 24 November, 2014; originally announced November 2014.

Bayesian Inference of Whole-Brain Networks

Authors: M. Hinne, T. Heskes, M. A. J. van Gerven

Abstract: In structural brain networks the connections of interest consist of white-matter fibre bundles between spatially segregated brain regions. The presence, location and orientation of these white matter tracts can be derived using diffusion MRI in combination with probabilistic tractography. Unfortunately, as of yet no approaches have been suggested that provide an undisputed way of inferring brain n… ▽ More In structural brain networks the connections of interest consist of white-matter fibre bundles between spatially segregated brain regions. The presence, location and orientation of these white matter tracts can be derived using diffusion MRI in combination with probabilistic tractography. Unfortunately, as of yet no approaches have been suggested that provide an undisputed way of inferring brain networks from tractography. In this paper, we provide a computational framework which we refer to as Bayesian connectomics. Rather than applying an arbitrary threshold to obtain a single network, we consider the posterior distribution of networks that are supported by the data, combined with an exponential random graph (ERGM) prior that captures a priori knowledge concerning the graph-theoretical properties of whole-brain networks. We show that, on simulated probabilistic tractography data, our approach is able to reconstruct whole-brain networks. In addition, our approach directly supports multi-model data fusion and group-level network inference. △ Less

Submitted 8 February, 2012; originally announced February 2012.

Comments: 10 pages, 2 figures