-
Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Models
Authors:
Luca M. Schulze Buschoff,
Konstantinos Voudouris,
Elif Akata,
Matthias Bethge,
Joshua B. Tenenbaum,
Eric Schulz
Abstract:
Pre-trained vision language models still fall short of human visual cognition. In an effort to improve visual cognition and align models with human behavior, we introduce visual stimuli and human judgments on visual cognition tasks, allowing us to systematically evaluate performance across cognitive domains under a consistent environment. We fine-tune models on ground truth data for intuitive phys…
▽ More
Pre-trained vision language models still fall short of human visual cognition. In an effort to improve visual cognition and align models with human behavior, we introduce visual stimuli and human judgments on visual cognition tasks, allowing us to systematically evaluate performance across cognitive domains under a consistent environment. We fine-tune models on ground truth data for intuitive physics and causal reasoning and find that this improves model performance in the respective fine-tuning domain. Furthermore, it can improve model alignment with human behavior. However, we find that task-specific fine-tuning does not contribute to robust human-like generalization to data with other visual characteristics or to tasks in other cognitive domains.
△ Less
Submitted 30 May, 2025; v1 submitted 21 February, 2025;
originally announced February 2025.
-
Centaur: a foundation model of human cognition
Authors:
Marcel Binz,
Elif Akata,
Matthias Bethge,
Franziska Brändle,
Fred Callaway,
Julian Coda-Forno,
Peter Dayan,
Can Demircan,
Maria K. Eckstein,
Noémi Éltető,
Thomas L. Griffiths,
Susanne Haridi,
Akshay K. Jagadish,
Li Ji-An,
Alexander Kipnis,
Sreejan Kumar,
Tobias Ludwig,
Marvin Mathony,
Marcelo Mattar,
Alireza Modirshanechi,
Surabhi S. Nath,
Joshua C. Peterson,
Milena Rmus,
Evan M. Russek,
Tankred Saanum
, et al. (15 additional authors not shown)
Abstract:
Establishing a unified theory of cognition has been a major goal of psychology. While there have been previous attempts to instantiate such theories by building computational models, we currently do not have one model that captures the human mind in its entirety. A first step in this direction is to create a model that can predict human behavior in a wide range of settings. Here we introduce Centa…
▽ More
Establishing a unified theory of cognition has been a major goal of psychology. While there have been previous attempts to instantiate such theories by building computational models, we currently do not have one model that captures the human mind in its entirety. A first step in this direction is to create a model that can predict human behavior in a wide range of settings. Here we introduce Centaur, a computational model that can predict and simulate human behavior in any experiment expressible in natural language. We derived Centaur by finetuning a state-of-the-art language model on a novel, large-scale data set called Psych-101. Psych-101 reaches an unprecedented scale, covering trial-by-trial data from over 60,000 participants performing over 10,000,000 choices in 160 experiments. Centaur not only captures the behavior of held-out participants better than existing cognitive models, but also generalizes to new cover stories, structural task modifications, and entirely new domains. Furthermore, we find that the model's internal representations become more aligned with human neural activity after finetuning. Taken together, our results demonstrate that it is possible to discover computational models that capture human behavior across a wide range of domains. We believe that such models provide tremendous potential for guiding the development of cognitive theories and present a case study to demonstrate this.
△ Less
Submitted 28 April, 2025; v1 submitted 26 October, 2024;
originally announced October 2024.
-
Visual cognition in multimodal large language models
Authors:
Luca M. Schulze Buschoff,
Elif Akata,
Matthias Bethge,
Eric Schulz
Abstract:
A chief goal of artificial intelligence is to build machines that think like people. Yet it has been argued that deep neural network architectures fail to accomplish this. Researchers have asserted these models' limitations in the domains of causal reasoning, intuitive physics, and intuitive psychology. Yet recent advancements, namely the rise of large language models, particularly those designed…
▽ More
A chief goal of artificial intelligence is to build machines that think like people. Yet it has been argued that deep neural network architectures fail to accomplish this. Researchers have asserted these models' limitations in the domains of causal reasoning, intuitive physics, and intuitive psychology. Yet recent advancements, namely the rise of large language models, particularly those designed for visual processing, have rekindled interest in the potential to emulate human-like cognitive abilities. This paper evaluates the current state of vision-based large language models in the domains of intuitive physics, causal reasoning, and intuitive psychology. Through a series of controlled experiments, we investigate the extent to which these modern models grasp complex physical interactions, causal relationships, and intuitive understanding of others' preferences. Our findings reveal that, while some of these models demonstrate a notable proficiency in processing and interpreting visual data, they still fall short of human capabilities in these areas. Our results emphasize the need for integrating more robust mechanisms for understanding causality, physical dynamics, and social cognition into modern-day, vision-based language models, and point out the importance of cognitively-inspired benchmarks.
△ Less
Submitted 8 August, 2024; v1 submitted 27 November, 2023;
originally announced November 2023.
-
Playing repeated games with Large Language Models
Authors:
Elif Akata,
Lion Schulz,
Julian Coda-Forno,
Seong Joon Oh,
Matthias Bethge,
Eric Schulz
Abstract:
LLMs are increasingly used in applications where they interact with humans and other agents. We propose to use behavioural game theory to study LLM's cooperation and coordination behaviour. We let different LLMs play finitely repeated $2\times2$ games with each other, with human-like strategies, and actual human players. Our results show that LLMs perform particularly well at self-interested games…
▽ More
LLMs are increasingly used in applications where they interact with humans and other agents. We propose to use behavioural game theory to study LLM's cooperation and coordination behaviour. We let different LLMs play finitely repeated $2\times2$ games with each other, with human-like strategies, and actual human players. Our results show that LLMs perform particularly well at self-interested games like the iterated Prisoner's Dilemma family. However, they behave sub-optimally in games that require coordination, like the Battle of the Sexes. We verify that these behavioural signatures are stable across robustness checks. We additionally show how GPT-4's behaviour can be modulated by providing additional information about its opponent and by using a "social chain-of-thought" (SCoT) strategy. This also leads to better scores and more successful coordination when interacting with human players. These results enrich our understanding of LLM's social behaviour and pave the way for a behavioural game theory for machines.
△ Less
Submitted 7 May, 2025; v1 submitted 26 May, 2023;
originally announced May 2023.
