-
Mutual Information Rate -- Linear Noise Approximation and Exact Computation
Authors:
Manuel Reinhardt,
Age J. Tjalma,
Anne-Lena Moor,
Christoph Zechner,
Pieter Rein ten Wolde
Abstract:
Efficient information processing is crucial for both living organisms and engineered systems. The mutual information rate, a core concept of information theory, quantifies the amount of information shared between the trajectories of input and output signals, and enables the quantification of information flow in dynamic systems. A common approach for estimating the mutual information rate is the Ga…
▽ More
Efficient information processing is crucial for both living organisms and engineered systems. The mutual information rate, a core concept of information theory, quantifies the amount of information shared between the trajectories of input and output signals, and enables the quantification of information flow in dynamic systems. A common approach for estimating the mutual information rate is the Gaussian approximation which assumes that the input and output trajectories follow Gaussian statistics. However, this method is limited to linear systems, and its accuracy in nonlinear or discrete systems remains unclear. In this work, we assess the accuracy of the Gaussian approximation for non-Gaussian systems by leveraging Path Weight Sampling (PWS), a recent technique for exactly computing the mutual information rate. In two case studies, we examine the limitations of the Gaussian approximation. First, we focus on discrete linear systems and demonstrate that, even when the system's statistics are nearly Gaussian, the Gaussian approximation fails to accurately estimate the mutual information rate. Second, we explore a continuous diffusive system with a nonlinear transfer function, revealing significant deviations between the Gaussian approximation and the exact mutual information rate as nonlinearity increases. Our results provide a quantitative evaluation of the Gaussian approximation's performance across different stochastic models and highlight when more computationally intensive methods, such as PWS, are necessary.
△ Less
Submitted 28 August, 2025;
originally announced August 2025.
-
ML-PWS: Estimating the Mutual Information Between Experimental Time Series Using Neural Networks
Authors:
Manuel Reinhardt,
Gašper Tkačik,
Pieter Rein ten Wolde
Abstract:
The ability to quantify information transmission is crucial for the analysis and design of natural and engineered systems. The information transmission rate is the fundamental measure for systems with time-varying signals, yet computing it is extremely challenging. In particular, the rate cannot be obtained directly from experimental time-series data without approximations, because of the high dim…
▽ More
The ability to quantify information transmission is crucial for the analysis and design of natural and engineered systems. The information transmission rate is the fundamental measure for systems with time-varying signals, yet computing it is extremely challenging. In particular, the rate cannot be obtained directly from experimental time-series data without approximations, because of the high dimensionality of the signal trajectory space. Path Weight Sampling (PWS) is a computational technique that makes it possible to obtain the information rate exactly for any stochastic system. However, it requires a mathematical model of the system of interest, be it described by a master equation or a set of differential equations. Here, we present a technique that employs Machine Learning (ML) to develop a generative model from experimental time-series data, which is then combined with PWS to obtain the information rate. We demonstrate the accuracy of this technique, called ML-PWS, by comparing its results on synthetic time-series data generated from a non-linear model against ground-truth results obtained by applying PWS directly to the same model. We illustrate the utility of ML-PWS by applying it to neuronal time-series data.
△ Less
Submitted 25 September, 2025; v1 submitted 22 August, 2025;
originally announced August 2025.
-
State- versus Reaction-Based Information Processing in Biochemical Networks
Authors:
Anne-Lena Moor,
Age Tjalma,
Manuel Reinhardt,
Pieter Rein ten Wolde,
Christoph Zechner
Abstract:
Trajectory mutual information is frequently used to quantify information transfer in biochemical systems. Tractable solutions of the trajectory mutual information can be obtained via the widely used Linear-Noise Approximation (LNA) using Gaussian channel theory. This approach is expected to be accurate for sufficiently large systems. However, recent observations show that there are cases, where th…
▽ More
Trajectory mutual information is frequently used to quantify information transfer in biochemical systems. Tractable solutions of the trajectory mutual information can be obtained via the widely used Linear-Noise Approximation (LNA) using Gaussian channel theory. This approach is expected to be accurate for sufficiently large systems. However, recent observations show that there are cases, where the mutual information obtained this way differs qualitatively from results derived using an exact Markov jump process formalism, and that the differences persist even for large systems. In this letter, we show that these differences can be explained by introducing the notion of reaction- versus state-based descriptions of trajectories. In chemical systems, the information is encoded in the sequence of reaction events, and the reaction-based trajectories of Markov jump processes capture this information. In contrast, the commonly used form of the LNA uses a state (concentration) based description of trajectories, which contains, in general, less information than a reaction-based description. Here, we show that an alternative formulation of the LNA that retains the reaction-specific information of trajectories can accurately describe the trajectory mutual information for large systems. We illustrate the consequences of different trajectory descriptions for two common cellular reaction motifs and discuss the connection with Berg-Purcell and Maximum-Likelihood sensing.
△ Less
Submitted 19 May, 2025;
originally announced May 2025.
-
Path Weight Sampling: Exact Monte Carlo Computation of the Mutual Information between Stochastic Trajectories
Authors:
Manuel Reinhardt,
Gašper Tkačik,
Pieter Rein ten Wolde
Abstract:
Most natural and engineered information-processing systems transmit information via signals that vary in time. Computing the information transmission rate or the information encoded in the temporal characteristics of these signals, requires the mutual information between the input and output signals as a function of time, i.e. between the input and output trajectories. Yet, this is notoriously dif…
▽ More
Most natural and engineered information-processing systems transmit information via signals that vary in time. Computing the information transmission rate or the information encoded in the temporal characteristics of these signals, requires the mutual information between the input and output signals as a function of time, i.e. between the input and output trajectories. Yet, this is notoriously difficult because of the high-dimensional nature of the trajectory space, and all existing techniques require approximations. We present an exact Monte Carlo technique called Path Weight Sampling (PWS) that, for the first time, makes it possible to compute the mutual information between input and output trajectories for any stochastic system that is described by a master equation. The principal idea is to use the master equation to evaluate the exact conditional probability of an individual output trajectory for a given input trajectory, and average this via Monte Carlo sampling in trajectory space to obtain the mutual information. We present three variants of PWS, which all generate the trajectories using the standard stochastic simulation algorithm. While Direct PWS is a brute-force method, Rosenbluth-Rosenbluth PWS exploits the analogy between signal trajectory sampling and polymer sampling, and Thermodynamic Integration PWS is based on a reversible work calculation in trajectory space. PWS also makes it possible to compute the mutual information between input and output trajectories for systems with hidden internal states as well as systems with feedback from output to input. Applying PWS to the bacterial chemotaxis system, consisting of 182 coupled chemical reactions, demonstrates not only that the scheme is highly efficient, but also that the number of receptor clusters is much smaller than hitherto believed, while their size is much larger.
△ Less
Submitted 4 October, 2023; v1 submitted 7 March, 2022;
originally announced March 2022.
