-
New analytic formulae for memory and prediction functions in reservoir computers with time delays
Authors:
Peyton Mullarkey,
Sarah Marzen
Abstract:
Time delays increase the effective dimensionality of reservoirs, thus suggesting that time delays in reservoirs can enhance their performance, particularly their memory and prediction abilities. We find new closed-form expressions for memory and prediction functions of linear time-delayed reservoirs in terms of the power spectrum of the input and the reservoir transfer function. We confirm this re…
▽ More
Time delays increase the effective dimensionality of reservoirs, thus suggesting that time delays in reservoirs can enhance their performance, particularly their memory and prediction abilities. We find new closed-form expressions for memory and prediction functions of linear time-delayed reservoirs in terms of the power spectrum of the input and the reservoir transfer function. We confirm this relationship numerically for some time-delayed reservoirs using simulations, including when the reservoir can be linearized but is actually nonlinear. Finally, we use these closed-form formulae to address the utility of multiple time delays in linear reservoirs in order to perform memory and prediction, finding similar results to previous work on nonlinear reservoirs. We hope these closed-form formulae can be used to understand memory and predictive capabilities in time-delayed reservoirs.
△ Less
Submitted 25 March, 2025;
originally announced March 2025.
-
Cognitive biases can move opinion dynamics from consensus to signatures of transient chaos
Authors:
Emily Dong,
Sarah Marzen
Abstract:
Interest in how democracies form consensus has increased recently, with statistical physics and economics approaches both suggesting that there is convergence to a fixed point in belief networks, but with fluctuations in opinions when there are ``stubborn'' voters. We modify a model of opinion dynamics in which agents are fully Bayesian to account for two cognitive biases: confirmation bias and in…
▽ More
Interest in how democracies form consensus has increased recently, with statistical physics and economics approaches both suggesting that there is convergence to a fixed point in belief networks, but with fluctuations in opinions when there are ``stubborn'' voters. We modify a model of opinion dynamics in which agents are fully Bayesian to account for two cognitive biases: confirmation bias and in-group bias. Confirmation bias occurs when the received information is considered to be more likely when it aligns with the receiver's beliefs. In-group bias occurs when the receiver further considers the information to be more likely when the receiver's beliefs and the sender's beliefs are aligned. We find that when there are no cognitive biases, a network of agents always converges to complete consensus. With confirmation bias alone, polarization can occur. With both biases present, consensus and polarization are possible, but when agents attempt to counteract confirmation bias, there can be signatures of transient chaos and ongoing opinion fluctuations. Based on this simple model, we conjecture that complex opinion fluctuations might be a generic feature of opinion dynamics when agents are Bayesian with biases.
△ Less
Submitted 29 May, 2024;
originally announced May 2024.
-
Learning about learning by many-body systems
Authors:
Weishun Zhong,
Jacob M. Gold,
Sarah Marzen,
Jeremy L. England,
Nicole Yunger Halpern
Abstract:
Diverse many-body systems, from soap bubbles to suspensions to polymers, learn and remember patterns in the drives that push them far from equilibrium. This learning may be leveraged for computation, memory, and engineering. Until now, many-body learning has been detected with thermodynamic properties, such as work absorption and strain. We progress beyond these macroscopic properties first define…
▽ More
Diverse many-body systems, from soap bubbles to suspensions to polymers, learn and remember patterns in the drives that push them far from equilibrium. This learning may be leveraged for computation, memory, and engineering. Until now, many-body learning has been detected with thermodynamic properties, such as work absorption and strain. We progress beyond these macroscopic properties first defined for equilibrium contexts: We quantify statistical mechanical learning using representation learning, a machine-learning model in which information squeezes through a bottleneck. By calculating properties of the bottleneck, we measure four facets of many-body systems' learning: classification ability, memory capacity, discrimination ability, and novelty detection. Numerical simulations of a classical spin glass illustrate our technique. This toolkit exposes self-organization that eludes detection by thermodynamic measures: Our toolkit more reliably and more precisely detects and quantifies learning by matter while providing a unifying framework for many-body learning.
△ Less
Submitted 22 October, 2021; v1 submitted 7 April, 2020;
originally announced April 2020.
-
Quantifying many-body learning far from equilibrium with representation learning
Authors:
Weishun Zhong,
Jacob M. Gold,
Sarah Marzen,
Jeremy L. England,
Nicole Yunger Halpern
Abstract:
Far-from-equilibrium many-body systems, from soap bubbles to suspensions to polymers, learn the drives that push them. This learning has been observed via thermodynamic properties, such as work absorption and strain. We move beyond these macroscopic properties that were first defined for equilibrium contexts: We quantify statistical mechanical learning with machine learning. Our toolkit relies on…
▽ More
Far-from-equilibrium many-body systems, from soap bubbles to suspensions to polymers, learn the drives that push them. This learning has been observed via thermodynamic properties, such as work absorption and strain. We move beyond these macroscopic properties that were first defined for equilibrium contexts: We quantify statistical mechanical learning with machine learning. Our toolkit relies on a structural parallel that we identify between far-from-equilibrium statistical mechanics and representation learning, which is undergone by neural networks that contain bottlenecks, including variational autoencoders. We train a variational autoencoder, via unsupervised learning, on configurations assumed by a many-body system during strong driving. We analyze the neural network's bottleneck to measure the many-body system's classification ability, memory capacity, discrimination ability, and novelty detection. Numerical simulations of a spin glass illustrate our technique. This toolkit exposes self-organization that eludes detection by thermodynamic measures, more reliably and more precisely identifying and quantifying learning by matter.
△ Less
Submitted 6 April, 2020; v1 submitted 10 January, 2020;
originally announced January 2020.
-
Statistical Signatures of Structural Organization: The case of long memory in renewal processes
Authors:
Sarah E. Marzen,
James P. Crutchfield
Abstract:
Identifying and quantifying memory are often critical steps in developing a mechanistic understanding of stochastic processes. These are particularly challenging and necessary when exploring processes that exhibit long-range correlations. The most common signatures employed rely on second-order temporal statistics and lead, for example, to identifying long memory in processes with power-law autoco…
▽ More
Identifying and quantifying memory are often critical steps in developing a mechanistic understanding of stochastic processes. These are particularly challenging and necessary when exploring processes that exhibit long-range correlations. The most common signatures employed rely on second-order temporal statistics and lead, for example, to identifying long memory in processes with power-law autocorrelation function and Hurst exponent greater than $1/2$. However, most stochastic processes hide their memory in higher-order temporal correlations. Information measures---specifically, divergences in the mutual information between a process' past and future (excess entropy) and minimal predictive memory stored in a process' causal states (statistical complexity)---provide a different way to identify long memory in processes with higher-order temporal correlations. However, there are no ergodic stationary processes with infinite excess entropy for which information measures have been compared to autocorrelation functions and Hurst exponents. Here, we show that fractal renewal processes---those with interevent distribution tails $\propto t^{-α}$---exhibit long memory via a phase transition at $α= 1$. Excess entropy diverges only there and statistical complexity diverges there and for all $α< 1$. When these processes do have power-law autocorrelation function and Hurst exponent greater than $1/2$, they do not have divergent excess entropy. This analysis breaks the intuitive association between these different quantifications of memory. We hope that the methods used here, based on causal states, provide some guide as to how to construct and analyze other long memory processes.
△ Less
Submitted 6 December, 2015;
originally announced December 2015.
-
The evolution of lossy compression
Authors:
Sarah E. Marzen,
Simon DeDeo
Abstract:
In complex environments, there are costs to both ignorance and perception. An organism needs to track fitness-relevant information about its world, but the more information it tracks, the more resources it must devote to memory and processing. Rate-distortion theory shows that, when errors are allowed, remarkably efficient internal representations can be found by biologically-plausible hill-climbi…
▽ More
In complex environments, there are costs to both ignorance and perception. An organism needs to track fitness-relevant information about its world, but the more information it tracks, the more resources it must devote to memory and processing. Rate-distortion theory shows that, when errors are allowed, remarkably efficient internal representations can be found by biologically-plausible hill-climbing mechanisms. We identify two regimes: a high-fidelity regime where perceptual costs scale logarithmically with environmental complexity, and a low-fidelity regime where perceptual costs are, remarkably, independent of the environment. When environmental complexity is rising, Darwinian evolution should drive organisms to the threshold between the high- and low-fidelity regimes. Organisms that code efficiently will find themselves able to make, just barely, the most subtle distinctions in their environment.
△ Less
Submitted 19 June, 2015;
originally announced June 2015.
-
An equivalence between a Maximum Caliber analysis of two-state kinetics and the Ising model
Authors:
Sarah Marzen,
Dave Wu,
Mandar Inamdar,
Rob Phillips
Abstract:
Application of the information-theoretic Maximum Caliber principle to the microtrajectories of a two-state system shows that the determination of key dynamical quantities can be mapped onto the evaluation of properties of the 1-D Ising model. The strategy described here is equivalent to an earlier Maximum Caliber formulation of the two-state problem, but reveals a different way of imposing the con…
▽ More
Application of the information-theoretic Maximum Caliber principle to the microtrajectories of a two-state system shows that the determination of key dynamical quantities can be mapped onto the evaluation of properties of the 1-D Ising model. The strategy described here is equivalent to an earlier Maximum Caliber formulation of the two-state problem, but reveals a different way of imposing the constraints which determine the probability distribution of allowed microtrajectories. The theoretical calculations of second moments, covariances, and correlation times that are obtained from Maximum Caliber agree well with simulated data of a particle diffusing on a double Gaussian surface, as well as with recent experiments on a particle trapped by a dual-well optical trap. The formalism reveals a new relationship between the average occupancy of the two states of the system, the average number of transitions between the two states that the system undergoes, Markov transition probabilities, and the discretization time step. In addition, Maxwell-like relations imply how measurements on one potential landscape can be used to make predictions about the dynamics on a different potential landscape, independent of further experiment.
△ Less
Submitted 16 August, 2010;
originally announced August 2010.
