-
A 7T fMRI dataset of synthetic images for out-of-distribution modeling of vision
Authors:
Alessandro T. Gifford,
Radoslaw M. Cichy,
Thomas Naselaris,
Kendrick Kay
Abstract:
Large-scale visual neural datasets such as the Natural Scenes Dataset (NSD) are boosting NeuroAI research by enabling computational models of the brain with performances beyond what was possible just a decade ago. However, these datasets lack out-of-distribution (OOD) components, which are crucial for the development of more robust models. Here, we address this limitation by releasing NSD-syntheti…
▽ More
Large-scale visual neural datasets such as the Natural Scenes Dataset (NSD) are boosting NeuroAI research by enabling computational models of the brain with performances beyond what was possible just a decade ago. However, these datasets lack out-of-distribution (OOD) components, which are crucial for the development of more robust models. Here, we address this limitation by releasing NSD-synthetic, a dataset consisting of 7T fMRI responses from the eight NSD subjects for 284 carefully controlled synthetic images. We show that NSD-synthetic's fMRI responses reliably encode stimulus-related information and are OOD with respect to NSD. Furthermore, OOD generalization tests on NSD-synthetic reveal differences between models of the brain that are not detected with NSD - specifically, self-supervised deep neural networks better explain neural responses than their task-supervised counterparts. These results showcase how NSD-synthetic enables OOD generalization tests that facilitate the development of more robust models of visual processing, and the formulation of more accurate theories of human vision.
△ Less
Submitted 8 March, 2025;
originally announced March 2025.
-
The Algonauts Project 2025 Challenge: How the Human Brain Makes Sense of Multimodal Movies
Authors:
Alessandro T. Gifford,
Domenic Bersch,
Marie St-Laurent,
Basile Pinsard,
Julie Boyle,
Lune Bellec,
Aude Oliva,
Gemma Roig,
Radoslaw M. Cichy
Abstract:
There is growing symbiosis between artificial and biological intelligence sciences: neural principles inspire new intelligent machines, which are in turn used to advance our theoretical understanding of the brain. To promote further collaboration between biological and artificial intelligence researchers, we introduce the 2025 edition of the Algonauts Project challenge: How the Human Brain Makes S…
▽ More
There is growing symbiosis between artificial and biological intelligence sciences: neural principles inspire new intelligent machines, which are in turn used to advance our theoretical understanding of the brain. To promote further collaboration between biological and artificial intelligence researchers, we introduce the 2025 edition of the Algonauts Project challenge: How the Human Brain Makes Sense of Multimodal Movies (https://algonautsproject.com/). In collaboration with the Courtois Project on Neuronal Modelling (CNeuroMod), this edition aims to bring forth a new generation of brain encoding models that are multimodal and that generalize well beyond their training distribution, by training them on the largest dataset of fMRI responses to movie watching available to date. Open to all, the 2025 challenge provides transparent, directly comparable results through a public leaderboard that is updated automatically after each submission to facilitate rapid model assessment and guide development. The challenge will end with a session at the 2025 Cognitive Computational Neuroscience (CCN) conference that will feature winning models. We welcome researchers interested in collaborating with the Algonauts Project by contributing ideas and datasets for future challenges.
△ Less
Submitted 6 January, 2025; v1 submitted 31 December, 2024;
originally announced January 2025.
-
In silico discovery of representational relationships across visual cortex
Authors:
Alessandro T. Gifford,
Maya A. Jastrzębowska,
Johannes J. D. Singer,
Radoslaw M. Cichy
Abstract:
Human vision is mediated by a complex interconnected network of cortical brain areas that jointly represent visual information. While these areas are increasingly understood in isolation, their representational relationships remain elusive. Here we developed relational neural control (RNC), and used it to investigate the representational relationships for univariate and multivariate fMRI responses…
▽ More
Human vision is mediated by a complex interconnected network of cortical brain areas that jointly represent visual information. While these areas are increasingly understood in isolation, their representational relationships remain elusive. Here we developed relational neural control (RNC), and used it to investigate the representational relationships for univariate and multivariate fMRI responses of areas across visual cortex. Through RNC we generated and explored in silico fMRI responses for large amounts of images, discovering controlling images that align or disentangle responses across areas, thus indicating their shared or unique representational content. This revealed a typical network-level configuration of representational relationships in which shared or unique representational content varied based on cortical distance, categorical selectivity, and position within the visual hierarchy. Closing the empirical cycle, we validated the in silico discoveries on in vivo fMRI responses from independent subjects. Together, this reveals how visual areas jointly represent the world as an interconnected network.
△ Less
Submitted 30 April, 2025; v1 submitted 16 November, 2024;
originally announced November 2024.
-
End-to-end topographic networks as models of cortical map formation and human visual behaviour: moving beyond convolutions
Authors:
Zejin Lu,
Adrien Doerig,
Victoria Bosch,
Bas Krahmer,
Daniel Kaiser,
Radoslaw M Cichy,
Tim C Kietzmann
Abstract:
Computational models are an essential tool for understanding the origin and functions of the topographic organisation of the primate visual system. Yet, vision is most commonly modelled by convolutional neural networks that ignore topography by learning identical features across space. Here, we overcome this limitation by developing All-Topographic Neural Networks (All-TNNs). Trained on visual inp…
▽ More
Computational models are an essential tool for understanding the origin and functions of the topographic organisation of the primate visual system. Yet, vision is most commonly modelled by convolutional neural networks that ignore topography by learning identical features across space. Here, we overcome this limitation by developing All-Topographic Neural Networks (All-TNNs). Trained on visual input, several features of primate topography emerge in All-TNNs: smooth orientation maps and cortical magnification in their first layer, and category-selective areas in their final layer. In addition, we introduce a novel dataset of human spatial biases in object recognition, which enables us to directly link models to behaviour. We demonstrate that All-TNNs significantly better align with human behaviour than previous state-of-the-art convolutional models due to their topographic nature. All-TNNs thereby mark an important step forward in understanding the spatial organisation of the visual brain and how it mediates visual behaviour.
△ Less
Submitted 18 August, 2023;
originally announced August 2023.
-
The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenes
Authors:
A. T. Gifford,
B. Lahner,
S. Saba-Sadiya,
M. G. Vilas,
A. Lascelles,
A. Oliva,
K. Kay,
G. Roig,
R. M. Cichy
Abstract:
The sciences of biological and artificial intelligence are ever more intertwined. Neural computational principles inspire new intelligent machines, which are in turn used to advance theoretical understanding of the brain. To promote further exchange of ideas and collaboration between biological and artificial intelligence researchers, we introduce the 2023 installment of the Algonauts Project chal…
▽ More
The sciences of biological and artificial intelligence are ever more intertwined. Neural computational principles inspire new intelligent machines, which are in turn used to advance theoretical understanding of the brain. To promote further exchange of ideas and collaboration between biological and artificial intelligence researchers, we introduce the 2023 installment of the Algonauts Project challenge: How the Human Brain Makes Sense of Natural Scenes (http://algonauts.csail.mit.edu). This installment prompts the fields of artificial and biological intelligence to come together towards building computational models of the visual brain using the largest and richest dataset of fMRI responses to visual scenes, the Natural Scenes Dataset (NSD). NSD provides high-quality fMRI responses to ~73,000 different naturalistic colored scenes, making it the ideal candidate for data-driven model building approaches promoted by the 2023 challenge. The challenge is open to all and makes results directly comparable and transparent through a public leaderboard automatically updated after each submission, thus allowing for rapid model development. We believe that the 2023 installment will spark symbiotic collaborations between biological and artificial intelligence scientists, leading to a deeper understanding of the brain through cutting-edge computational models and to novel ways of engineering artificial intelligent agents through inductive biases from biological systems.
△ Less
Submitted 11 July, 2023; v1 submitted 9 January, 2023;
originally announced January 2023.
-
Net2Brain: A Toolbox to compare artificial vision models with human brain responses
Authors:
Domenic Bersch,
Kshitij Dwivedi,
Martina Vilas,
Radoslaw M. Cichy,
Gemma Roig
Abstract:
We introduce Net2Brain, a graphical and command-line user interface toolbox for comparing the representational spaces of artificial deep neural networks (DNNs) and human brain recordings. While different toolboxes facilitate only single functionalities or only focus on a small subset of supervised image classification models, Net2Brain allows the extraction of activations of more than 600 DNNs tra…
▽ More
We introduce Net2Brain, a graphical and command-line user interface toolbox for comparing the representational spaces of artificial deep neural networks (DNNs) and human brain recordings. While different toolboxes facilitate only single functionalities or only focus on a small subset of supervised image classification models, Net2Brain allows the extraction of activations of more than 600 DNNs trained to perform a diverse range of vision-related tasks (e.g semantic segmentation, depth estimation, action recognition, etc.), over both image and video datasets. The toolbox computes the representational dissimilarity matrices (RDMs) over those activations and compares them to brain recordings using representational similarity analysis (RSA), weighted RSA, both in specific ROIs and with searchlight search. In addition, it is possible to add a new data set of stimuli and brain recordings to the toolbox for evaluation. We demonstrate the functionality and advantages of Net2Brain with an example showcasing how it can be used to test hypotheses of cognitive computational neuroscience.
△ Less
Submitted 25 August, 2022; v1 submitted 20 August, 2022;
originally announced August 2022.
-
The Algonauts Project 2021 Challenge: How the Human Brain Makes Sense of a World in Motion
Authors:
R. M. Cichy,
K. Dwivedi,
B. Lahner,
A. Lascelles,
P. Iamshchinina,
M. Graumann,
A. Andonian,
N. A. R. Murty,
K. Kay,
G. Roig,
A. Oliva
Abstract:
The sciences of natural and artificial intelligence are fundamentally connected. Brain-inspired human-engineered AI are now the standard for predicting human brain responses during vision, and conversely, the brain continues to inspire invention in AI. To promote even deeper connections between these fields, we here release the 2021 edition of the Algonauts Project Challenge: How the Human Brain M…
▽ More
The sciences of natural and artificial intelligence are fundamentally connected. Brain-inspired human-engineered AI are now the standard for predicting human brain responses during vision, and conversely, the brain continues to inspire invention in AI. To promote even deeper connections between these fields, we here release the 2021 edition of the Algonauts Project Challenge: How the Human Brain Makes Sense of a World in Motion (http://algonauts.csail.mit.edu/). We provide whole-brain fMRI responses recorded while 10 human participants viewed a rich set of over 1,000 short video clips depicting everyday events. The goal of the challenge is to accurately predict brain responses to these video clips. The format of our challenge ensures rapid development, makes results directly comparable and transparent, and is open to all. In this way it facilitates interdisciplinary collaboration towards a common goal of understanding visual intelligence. The 2021 Algonauts Project is conducted in collaboration with the Cognitive Computational Neuroscience (CCN) conference.
△ Less
Submitted 28 April, 2021;
originally announced April 2021.
-
The Algonauts Project: A Platform for Communication between the Sciences of Biological and Artificial Intelligence
Authors:
Radoslaw Martin Cichy,
Gemma Roig,
Alex Andonian,
Kshitij Dwivedi,
Benjamin Lahner,
Alex Lascelles,
Yalda Mohsenzadeh,
Kandan Ramakrishnan,
Aude Oliva
Abstract:
In the last decade, artificial intelligence (AI) models inspired by the brain have made unprecedented progress in performing real-world perceptual tasks like object classification and speech recognition. Recently, researchers of natural intelligence have begun using those AI models to explore how the brain performs such tasks. These developments suggest that future progress will benefit from incre…
▽ More
In the last decade, artificial intelligence (AI) models inspired by the brain have made unprecedented progress in performing real-world perceptual tasks like object classification and speech recognition. Recently, researchers of natural intelligence have begun using those AI models to explore how the brain performs such tasks. These developments suggest that future progress will benefit from increased interaction between disciplines. Here we introduce the Algonauts Project as a structured and quantitative communication channel for interdisciplinary interaction between natural and artificial intelligence researchers. The project's core is an open challenge with a quantitative benchmark whose goal is to account for brain data through computational models. This project has the potential to provide better models of natural intelligence and to gather findings that advance AI. The 2019 Algonauts Project focuses on benchmarking computational models predicting human brain activity when people look at pictures of objects. The 2019 edition of the Algonauts Project is available online: http://algonauts.csail.mit.edu/.
△ Less
Submitted 14 May, 2019;
originally announced May 2019.
-
Recurrence is required to capture the representational dynamics of the human visual system
Authors:
Tim C Kietzmann,
Courtney J Spoerer,
Lynn Sörensen,
Radoslaw M Cichy,
Olaf Hauk,
Nikolaus Kriegeskorte
Abstract:
The human visual system is an intricate network of brain regions that enables us to recognize the world around us. Despite its abundant lateral and feedback connections, object processing is commonly viewed and studied as a feedforward process. Here, we measure and model the rapid representational dynamics across multiple stages of the human ventral stream using time-resolved brain imaging and dee…
▽ More
The human visual system is an intricate network of brain regions that enables us to recognize the world around us. Despite its abundant lateral and feedback connections, object processing is commonly viewed and studied as a feedforward process. Here, we measure and model the rapid representational dynamics across multiple stages of the human ventral stream using time-resolved brain imaging and deep learning. We observe substantial representational transformations during the first 300 ms of processing within and across ventral-stream regions. Categorical divisions emerge in sequence, cascading forward and in reverse across regions, and Granger causality analysis suggests bidirectional information flow between regions. Finally, recurrent deep neural network models clearly outperform parameter-matched feedforward models in terms of their ability to capture the multi-region cortical dynamics. Targeted virtual cooling experiments on the recurrent deep network models further substantiate the importance of their lateral and top-down connections. These results establish that recurrent models are required to understand information processing in the human ventral stream.
△ Less
Submitted 8 October, 2019; v1 submitted 14 March, 2019;
originally announced March 2019.
-
Deep Neural Networks predict Hierarchical Spatio-temporal Cortical Dynamics of Human Visual Object Recognition
Authors:
Radoslaw M. Cichy,
Aditya Khosla,
Dimitrios Pantazis,
Antonio Torralba,
Aude Oliva
Abstract:
The complex multi-stage architecture of cortical visual pathways provides the neural basis for efficient visual object recognition in humans. However, the stage-wise computations therein remain poorly understood. Here, we compared temporal (magnetoencephalography) and spatial (functional MRI) visual brain representations with representations in an artificial deep neural network (DNN) tuned to the…
▽ More
The complex multi-stage architecture of cortical visual pathways provides the neural basis for efficient visual object recognition in humans. However, the stage-wise computations therein remain poorly understood. Here, we compared temporal (magnetoencephalography) and spatial (functional MRI) visual brain representations with representations in an artificial deep neural network (DNN) tuned to the statistics of real-world visual recognition. We showed that the DNN captured the stages of human visual processing in both time and space from early visual areas towards the dorsal and ventral streams. Further investigation of crucial DNN parameters revealed that while model architecture was important, training on real-world categorization was necessary to enforce spatio-temporal hierarchical relationships with the brain. Together our results provide an algorithmically informed view on the spatio-temporal dynamics of visual object recognition in the human visual brain.
△ Less
Submitted 12 January, 2016;
originally announced January 2016.
-
Can visual information encoded in cortical columns be decoded from magnetoencephalography data in humans?
Authors:
Radoslaw Martin Cichy,
Dimitrios Pantazis
Abstract:
It is a principal open question whether noninvasive imaging methods in humans can decode information encoded at a spatial scale as fine as the basic functional unit of cortex: cortical columns. We addressed this question in five magnetoencephalography (MEG) experiments by investigating the encoding of a columnar-level encoded visual feature: contrast edge orientation. We found that MEG signals con…
▽ More
It is a principal open question whether noninvasive imaging methods in humans can decode information encoded at a spatial scale as fine as the basic functional unit of cortex: cortical columns. We addressed this question in five magnetoencephalography (MEG) experiments by investigating the encoding of a columnar-level encoded visual feature: contrast edge orientation. We found that MEG signals contained orientation-specific information as early as ~50ms after stimulus onset even when controlling for confounds, such as overrepresentation of particular orientations, stimulus edge interactions, and global form-related signals. Theoretical modeling confirmed the plausibility of this empirical result. An essential consequence of our results is that information encoded in the human brain at the level of cortical columns should in general be accessible by multivariate analysis of electrophysiological signals.
△ Less
Submitted 27 March, 2015;
originally announced March 2015.
