-
Complex behavior from intrinsic motivation to occupy action-state path space
Authors:
Jorge Ramírez-Ruiz,
Dmytro Grytskyy,
Chiara Mastrogiuseppe,
Yamen Habib,
Rubén Moreno-Bote
Abstract:
Most theories of behavior posit that agents tend to maximize some form of reward or utility. However, animals very often move with curiosity and seem to be motivated in a reward-free manner. Here we abandon the idea of reward maximization, and propose that the goal of behavior is maximizing occupancy of future paths of actions and states. According to this maximum occupancy principle, rewards are…
▽ More
Most theories of behavior posit that agents tend to maximize some form of reward or utility. However, animals very often move with curiosity and seem to be motivated in a reward-free manner. Here we abandon the idea of reward maximization, and propose that the goal of behavior is maximizing occupancy of future paths of actions and states. According to this maximum occupancy principle, rewards are the means to occupy path space, not the goal per se; goal-directedness simply emerges as rational ways of searching for resources so that movement, understood amply, never ends. We find that action-state path entropy is the only measure consistent with additivity and other intuitive properties of expected future action-state path occupancy. We provide analytical expressions that relate the optimal policy and state-value function, and prove convergence of our value iteration algorithm. Using discrete and continuous state tasks, including a high--dimensional controller, we show that complex behaviors such as `dancing', hide-and-seek and a basic form of altruistic behavior naturally result from the intrinsic motivation to occupy path space. All in all, we present a theory of behavior that generates both variability and goal-directedness in the absence of reward maximization.
△ Less
Submitted 24 February, 2024; v1 submitted 20 May, 2022;
originally announced May 2022.
-
Deep imagination is a close to optimal policy for planning in large decision trees under limited resources
Authors:
Ruben Moreno-Bote,
Chiara Mastrogiuseppe
Abstract:
Many decisions involve choosing an uncertain course of actions in deep and wide decision trees, as when we plan to visit an exotic country for vacation. In these cases, exhaustive search for the best sequence of actions is not tractable due to the large number of possibilities and limited time or computational resources available to make the decision. Therefore, planning agents need to balance bre…
▽ More
Many decisions involve choosing an uncertain course of actions in deep and wide decision trees, as when we plan to visit an exotic country for vacation. In these cases, exhaustive search for the best sequence of actions is not tractable due to the large number of possibilities and limited time or computational resources available to make the decision. Therefore, planning agents need to balance breadth (exploring many actions at each level of the tree) and depth (exploring many levels in the tree) to allocate optimally their finite search capacity. We provide efficient analytical solutions and numerical analysis to the problem of allocating finite sampling capacity in one shot to large decision trees. We find that in general the optimal policy is to allocate few samples per level so that deep levels can be reached, thus favoring depth over breadth search. In contrast, in poor environments and at low capacity, it is best to broadly sample branches at the cost of not sampling deeply, although this policy is marginally better than deep allocations. Our results provide a theoretical foundation for the optimality of deep imagination for planning and show that it is a generally valid heuristic that could have evolved from the finite constraints of cognitive systems.
△ Less
Submitted 13 April, 2021;
originally announced April 2021.
-
Optimal allocation of finite sampling capacity in accumulator models of multi-alternative decision making
Authors:
Jorge Ramírez-Ruiz,
Rubén Moreno-Bote
Abstract:
When facing many options, we narrow down our focus to very few of them. Although behaviors like this can be a sign of heuristics, they can actually be optimal under limited cognitive resources. Here we study the problem of how to optimally allocate limited sampling time to multiple options, modelled as accumulators of noisy evidence, to determine the most profitable one. We show that the effective…
▽ More
When facing many options, we narrow down our focus to very few of them. Although behaviors like this can be a sign of heuristics, they can actually be optimal under limited cognitive resources. Here we study the problem of how to optimally allocate limited sampling time to multiple options, modelled as accumulators of noisy evidence, to determine the most profitable one. We show that the effective sampling capacity of an agent increases with both available time and the discriminability of the options, and optimal policies undergo a sharp transition as a function of it. For small capacity, it is best to allocate time evenly to exactly five options and to ignore all the others, regardless of the prior distribution of rewards. For large capacities, the optimal number of sampled accumulators grows sub-linearly, closely following a power law for a wide variety of priors. We find that allocating equal times to the sampled accumulators is better than using uneven time allocations. Our work highlights that multi-alternative decisions are endowed with breadth-depth tradeoffs, demonstrates how their optimal solutions depend on the amount of limited resources and the variability of the environment, and shows that narrowing down to a handful of options is always optimal for small capacities.
△ Less
Submitted 2 February, 2021;
originally announced February 2021.
-
Family of closed-form solutions for two-dimensional correlated diffusion processes
Authors:
Haozhe Shan,
Rubén Moreno-Bote,
Jan Drugowitsch
Abstract:
Diffusion processes with boundaries are models of transport phenomena with wide applicability across many fields. These processes are described by their probability density functions (PDFs), which often obey Fokker-Planck equations (FPEs). While obtaining analytical solutions is often possible in the absence of boundaries, obtaining closed-form solutions to the FPE is more challenging once absorbi…
▽ More
Diffusion processes with boundaries are models of transport phenomena with wide applicability across many fields. These processes are described by their probability density functions (PDFs), which often obey Fokker-Planck equations (FPEs). While obtaining analytical solutions is often possible in the absence of boundaries, obtaining closed-form solutions to the FPE is more challenging once absorbing boundaries are present. As a result, analyses of these processes have largely relied on approximations or direct simulations. In this paper, we studied two-dimensional, time-homogeneous, spatially-correlated diffusion with linear, axis-aligned, absorbing boundaries. Our main result is the explicit construction of a full family of closed-form solutions for their PDFs using the method of images (MoI). We found that such solutions can be built if and only if the correlation coefficient $ρ$ between the two diffusing processes takes one of a numerable set of values. Using a geometric argument, we derived the complete set of $ρ$'s where such solutions can be found. Solvable $ρ$'s are given by $ρ= - \cos \left( \fracπ{k} \right)$, where $k \in \mathbb{Z}^+ \cup \{ +\infty\}$. Solutions were validated in simulations. Qualitative behaviors of the process appear to vary smoothly over $ρ$, allowing extrapolation from our solutions to cases with unsolvable $ρ$'s.
△ Less
Submitted 4 September, 2019; v1 submitted 7 July, 2019;
originally announced July 2019.
-
What can neuronal populations tell us about cognition?
Authors:
Iñigo Arandia-Romero,
Ramon Nogueira,
Gabriela Mochol,
Rubén Moreno-Bote
Abstract:
Nowadays, it is possible to record the activity of hundreds of cells at the same time in behaving animals. However, these data are often treated and analyzed as if they consisted of many independently recorded neurons. How can neuronal populations be uniquely used to learn about cognition? We describe recent work that shows that populations of simultaneously recorded neurons are fundamental to und…
▽ More
Nowadays, it is possible to record the activity of hundreds of cells at the same time in behaving animals. However, these data are often treated and analyzed as if they consisted of many independently recorded neurons. How can neuronal populations be uniquely used to learn about cognition? We describe recent work that shows that populations of simultaneously recorded neurons are fundamental to understand the basis of decision-making, including processes such as ongoing deliberations and decision confidence, which generally fall outside the reach of single-cell analysis. Thus, neuronal population data allow addressing novel questions, but they also come with so far unsolved challenges.
△ Less
Submitted 4 November, 2017;
originally announced November 2017.
-
Multiplicative and additive modulation of neuronal tuning with population activity affects encoded information
Authors:
Iñigo Arandia-Romero,
Seiji Tanabe,
Jan Drugowitsch,
Adam Kohn,
Rubén Moreno-Bote
Abstract:
Numerous studies have shown that neuronal responses are modulated by stimulus properties, and also by the state of the local network. However, little is known about how activity fluctuations of neuronal populations modulate the sensory tuning of cells and affect their encoded information. We found that fluctuations in ongoing and stimulus-evoked population activity in primate visual cortex modulat…
▽ More
Numerous studies have shown that neuronal responses are modulated by stimulus properties, and also by the state of the local network. However, little is known about how activity fluctuations of neuronal populations modulate the sensory tuning of cells and affect their encoded information. We found that fluctuations in ongoing and stimulus-evoked population activity in primate visual cortex modulate the tuning of neurons in a multiplicative and additive manner. While distributed on a continuum, neurons with stronger multiplicative effects tended to have less additive modulation, and vice versa. The information encoded by multiplicatively-modulated neurons increased with greater population activity, while that of additively-modulated neurons decreased. These effects offset each other, so that population activity had little effect on total information. Our results thus suggest that intrinsic activity fluctuations may act as a "traffic light" that determines which subset of neurons are most informative.
△ Less
Submitted 4 November, 2017;
originally announced November 2017.
-
Theory of input spike auto- and cross-correlations and their effect on the response of spiking neurons
Authors:
Ruben Moreno-Bote,
Alfonso Renart,
Nestor Parga
Abstract:
Spike correlations between neurons are ubiquitous in the cortex, but their role is at present not understood. Here we describe the firing response of a leaky integrate-and-fire neuron (LIF) when it receives a temporarily correlated input generated by presynaptic correlated neuronal populations. Input correlations are characterized in terms of the firing rates, Fano factors, correlation coefficie…
▽ More
Spike correlations between neurons are ubiquitous in the cortex, but their role is at present not understood. Here we describe the firing response of a leaky integrate-and-fire neuron (LIF) when it receives a temporarily correlated input generated by presynaptic correlated neuronal populations. Input correlations are characterized in terms of the firing rates, Fano factors, correlation coefficients and correlation timescale of the neurons driving the target neuron. We show that the sum of the presynaptic spike trains cannot be well described by a Poisson process. Solutions of the output firing rate are found in the limit of short and long correlation time scales.
△ Less
Submitted 11 October, 2007;
originally announced October 2007.
-
Auto and crosscorrelograms for the spike response of LIF neurons with slow synapses
Authors:
Ruben Moreno-Bote,
Nestor Parga
Abstract:
An analytical description of the response properties of simple but realistic neuron models in the presence of noise is still lacking. We determine completely up to the second order the firing statistics of a single and a pair of leaky integrate-and-fire neurons (LIFs) receiving some common slowly filtered white noise. In particular, the auto- and cross-correlation functions of the output spike t…
▽ More
An analytical description of the response properties of simple but realistic neuron models in the presence of noise is still lacking. We determine completely up to the second order the firing statistics of a single and a pair of leaky integrate-and-fire neurons (LIFs) receiving some common slowly filtered white noise. In particular, the auto- and cross-correlation functions of the output spike trains of pairs of cells are obtained from an improvement of the adiabatic approximation introduced in \cite{Mor+04}. These two functions define the firing variability and firing synchronization between neurons, and are of much importance for understanding neuron communication.
△ Less
Submitted 11 October, 2007;
originally announced October 2007.
