-
A scalable estimator of high-order information in complex dynamical systems
Authors:
Alberto Liardi,
George Blackburne,
Hardik Rajpal,
Fernando E. Rosas,
Pedro A. M. Mediano
Abstract:
Our understanding of neural systems rests on our ability to characterise how they perform distributed computation and integrate information. Advances in information theory have introduced several quantities to describe complex information structures, where collective patterns of coordination emerge from high-order (i.e. beyond-pairwise) interdependencies. Unfortunately, the use of these approaches…
▽ More
Our understanding of neural systems rests on our ability to characterise how they perform distributed computation and integrate information. Advances in information theory have introduced several quantities to describe complex information structures, where collective patterns of coordination emerge from high-order (i.e. beyond-pairwise) interdependencies. Unfortunately, the use of these approaches to study large neural systems is severely hindered by the poor scalability of existing techniques. Moreover, there are relatively few measures specifically designed for multivariate time series data. Here we introduce a novel measure of information about macroscopic structures, termed M-information, which quantifies the high-order integration of information in complex dynamical systems. We show that M-information can be calculated via a convex optimisation problem, and we derive a robust and efficient algorithm that scales gracefully with system size. Our analyses show that M-information is resilient to noise, indexes critical behaviour in artificial neuronal populations, and reflects task performance in real-world mouse brain activity data. Furthermore, M-information can be incorporated into existing information decomposition frameworks to reveal a comprehensive taxonomy of information dynamics. Taken together, these results help us unravel collective computation in complex neural systems.
△ Less
Submitted 23 June, 2025;
originally announced June 2025.
-
Shannon invariants: A scalable approach to information decomposition
Authors:
Aaron J. Gutknecht,
Fernando E. Rosas,
David A. Ehrlich,
Abdullah Makkeh,
Pedro A. M. Mediano,
Michael Wibral
Abstract:
Distributed systems, such as biological and artificial neural networks, process information via complex interactions engaging multiple subsystems, resulting in high-order patterns with distinct properties across scales. Investigating how these systems process information remains challenging due to difficulties in defining appropriate multivariate metrics and ensuring their scalability to large sys…
▽ More
Distributed systems, such as biological and artificial neural networks, process information via complex interactions engaging multiple subsystems, resulting in high-order patterns with distinct properties across scales. Investigating how these systems process information remains challenging due to difficulties in defining appropriate multivariate metrics and ensuring their scalability to large systems. To address these challenges, we introduce a novel framework based on what we call "Shannon invariants" -- quantities that capture essential properties of high-order information processing in a way that depends only on the definition of entropy and can be efficiently calculated for large systems. Our theoretical results demonstrate how Shannon invariants can be used to resolve long-standing ambiguities regarding the interpretation of widely used multivariate information-theoretic measures. Moreover, our practical results reveal distinctive information-processing signatures of various deep learning architectures across layers, which lead to new insights into how these systems process information and how this evolves during training. Overall, our framework resolves fundamental limitations in analyzing high-order phenomena and offers broad opportunities for theoretical developments and empirical analyses.
△ Less
Submitted 22 April, 2025;
originally announced April 2025.
-
Null models for comparing information decomposition across complex systems
Authors:
Alberto Liardi,
Fernando E. Rosas,
Robin L. Carhart-Harris,
George Blackburne,
Daniel Bor,
Pedro A. M. Mediano
Abstract:
A key feature of information theory is its universality, as it can be applied to study a broad variety of complex systems. However, many information-theoretic measures can vary significantly even across systems with similar properties, making normalisation techniques essential for allowing meaningful comparisons across datasets. Inspired by the framework of Partial Information Decomposition (PID),…
▽ More
A key feature of information theory is its universality, as it can be applied to study a broad variety of complex systems. However, many information-theoretic measures can vary significantly even across systems with similar properties, making normalisation techniques essential for allowing meaningful comparisons across datasets. Inspired by the framework of Partial Information Decomposition (PID), here we introduce Null Models for Information Theory (NuMIT), a null model-based non-linear normalisation procedure which improves upon standard entropy-based normalisation approaches and overcomes their limitations. We provide practical implementations of the technique for systems with different statistics, and showcase the method on synthetic models and on human neuroimaging data. Our results demonstrate that NuMIT provides a robust and reliable tool to characterise complex systems of interest, allowing cross-dataset comparisons and providing a meaningful significance test for PID analyses.
△ Less
Submitted 15 October, 2024;
originally announced October 2024.
-
Characterising high-order interdependence via entropic conjugation
Authors:
Fernando E. Rosas,
Aaron Gutknecht,
Pedro A. M. Mediano,
Michael Gastpar
Abstract:
High-order phenomena play crucial roles in many systems of interest, but their analysis is often highly nontrivial. There is a rich literature providing a number of alternative information-theoretic quantities capturing high-order phenomena, but their interpretation and relationship with each other is not well understood. The lack of principles unifying these quantities obscures the choice of tool…
▽ More
High-order phenomena play crucial roles in many systems of interest, but their analysis is often highly nontrivial. There is a rich literature providing a number of alternative information-theoretic quantities capturing high-order phenomena, but their interpretation and relationship with each other is not well understood. The lack of principles unifying these quantities obscures the choice of tools for enabling specific type of analyses. Here we show how an entropic conjugation provides a theoretically grounded principle to investigate the space of possible high-order quantities, clarifying the nature of the existent metrics while revealing gaps in the literature. This leads to identify novel notions of symmetry and skew-symmetry as key properties for guaranteeing a balanced account of high-order interdependencies and enabling broadly applicable analyses across physical systems.
△ Less
Submitted 14 October, 2024;
originally announced October 2024.
-
The Fast Möbius Transform: An algebraic approach to information decomposition
Authors:
Abel Jansma,
Pedro A. M. Mediano,
Fernando E. Rosas
Abstract:
The partial information decomposition (PID) and its extension integrated information decomposition ($Φ$ID) are promising frameworks to investigate information phenomena involving multiple variables. An important limitation of these approaches is the high computational cost involved in their calculation. Here we leverage fundamental algebraic properties of these decompositions to enable a computati…
▽ More
The partial information decomposition (PID) and its extension integrated information decomposition ($Φ$ID) are promising frameworks to investigate information phenomena involving multiple variables. An important limitation of these approaches is the high computational cost involved in their calculation. Here we leverage fundamental algebraic properties of these decompositions to enable a computationally-efficient method to estimate them, which we call the fast Möbius transform. Our approach is based on a novel formula for estimating the Möbius function that circumvents important computational bottlenecks. We showcase the capabilities of this approach by presenting two analyses that would be unfeasible without this method: decomposing the information that neural activity at different frequency bands yield about the brain's macroscopic functional organisation, and identifying distinctive dynamical properties of the interactions between multiple voices in baroque music. Overall, our proposed approach illuminates the value of algebraic facets of information decomposition and opens the way to a wide range of future analyses.
△ Less
Submitted 8 October, 2024;
originally announced October 2024.
-
Biological arrow of time: Emergence of tangled information hierarchies and self-modelling dynamics
Authors:
Mikhail Prokopenko,
Paul C. W. Davies,
Michael Harré,
Marcus Heisler,
Zdenka Kuncic,
Geraint F. Lewis,
Ori Livson,
Joseph T. Lizier,
Fernando E. Rosas
Abstract:
We study open-ended evolution by focusing on computational and information-processing dynamics underlying major evolutionary transitions. In doing so, we consider biological organisms as hierarchical dynamical systems that generate regularities in their phase-spaces through interactions with their environment. These emergent information patterns can then be encoded within the organism's components…
▽ More
We study open-ended evolution by focusing on computational and information-processing dynamics underlying major evolutionary transitions. In doing so, we consider biological organisms as hierarchical dynamical systems that generate regularities in their phase-spaces through interactions with their environment. These emergent information patterns can then be encoded within the organism's components, leading to self-modelling "tangled hierarchies". Our main conjecture is that when macro-scale patterns are encoded within micro-scale components, it creates fundamental tensions (computational inconsistencies) between what is encodable at a particular evolutionary stage and what is potentially realisable in the environment. A resolution of these tensions triggers an evolutionary transition which expands the problem-space, at the cost of generating new tensions in the expanded space, in a continual process. We argue that biological complexification can be interpreted computation-theoretically, within the Gödel--Turing--Post recursion-theoretic framework, as open-ended generation of computational novelty. In general, this process can be viewed as a meta-simulation performed by higher-order systems that successively simulate the computation carried out by lower-order systems. This computation-theoretic argument provides a basis for hypothesising the biological arrow of time.
△ Less
Submitted 18 September, 2024;
originally announced September 2024.
-
Explosive neural networks via higher-order interactions in curved statistical manifolds
Authors:
Miguel Aguilera,
Pablo A. Morales,
Fernando E. Rosas,
Hideaki Shimazaki
Abstract:
Higher-order interactions underlie complex phenomena in systems such as biological and artificial neural networks, but their study is challenging due to the scarcity of tractable models. By leveraging a generalisation of the maximum entropy principle, here we introduce curved neural networks as a class of prototypical models with a limited number of parameters that are particularly well-suited for…
▽ More
Higher-order interactions underlie complex phenomena in systems such as biological and artificial neural networks, but their study is challenging due to the scarcity of tractable models. By leveraging a generalisation of the maximum entropy principle, here we introduce curved neural networks as a class of prototypical models with a limited number of parameters that are particularly well-suited for studying higher-order phenomena. Through exact mean-field descriptions, we show that these curved neural networks implement a self-regulating annealing process that can accelerate memory retrieval, leading to explosive order-disorder phase transitions with multi-stability and hysteresis effects. Moreover, by analytically exploring their memory-retrieval capacity using the replica trick near ferromagnetic and spin-glass phase boundaries, we demonstrate that these networks can enhance memory capacity and robustness of retrieval over classical associative-memory networks. Overall, the proposed framework provides parsimonious models amenable to analytical study, revealing novel higher-order phenomena in complex networks.
△ Less
Submitted 4 February, 2025; v1 submitted 5 August, 2024;
originally announced August 2024.
-
Characterising directed and undirected metrics of high-order interdependence
Authors:
Fernando E. Rosas,
Pedro A. M. Mediano,
Michael Gastpar
Abstract:
Systems of interest for theoretical or experimental work often exhibit high-order interactions, corresponding to statistical interdependencies in groups of variables that cannot be reduced to dependencies in subsets of them. While still under active development, the framework of partial information decomposition (PID) has emerged as the dominant approach to conceptualise and calculate high-order i…
▽ More
Systems of interest for theoretical or experimental work often exhibit high-order interactions, corresponding to statistical interdependencies in groups of variables that cannot be reduced to dependencies in subsets of them. While still under active development, the framework of partial information decomposition (PID) has emerged as the dominant approach to conceptualise and calculate high-order interdependencies. PID approaches can be grouped in two types: directed approaches that divide variables into sources and targets, and undirected approaches that treat all variables equally. Directed and undirected approaches are usually employed to investigate different scenarios, and hence little is known about how these two types of approaches may relate to each other, or if their corresponding quantities are linked in some way. In this paper we investigate the relationship between the redundancy-synergy index (RSI) and the O-information, which are practical metrics of directed and undirected high-order interdependencies, respectively. Our results reveal tight links between these two quantities, and provide interpretations of them in terms of likelihood ratios in a hypothesis testing setting, as well as in terms of projections in information geometry.
△ Less
Submitted 10 April, 2024;
originally announced April 2024.
-
Information decomposition reveals hidden high-order contributions to temporal irreversibility
Authors:
Andrea I Luppi,
Fernando E. Rosas,
Gustavo Deco,
Morten L. Kringelbach,
Pedro A. M. Mediano
Abstract:
Temporal irreversibility, often referred to as the arrow of time, is a fundamental concept in statistical mechanics. Markers of irreversibility also provide a powerful characterisation of information processing in biological systems. However, current approaches tend to describe temporal irreversibility in terms of a single scalar quantity, without disentangling the underlying dynamics that contrib…
▽ More
Temporal irreversibility, often referred to as the arrow of time, is a fundamental concept in statistical mechanics. Markers of irreversibility also provide a powerful characterisation of information processing in biological systems. However, current approaches tend to describe temporal irreversibility in terms of a single scalar quantity, without disentangling the underlying dynamics that contribute to irreversibility. Here we propose a broadly applicable information-theoretic framework to characterise the arrow of time in multivariate time series, which yields qualitatively different types of irreversible information dynamics. This multidimensional characterisation reveals previously unreported high-order modes of irreversibility, and establishes a formal connection between recent heuristic markers of temporal irreversibility and metrics of information processing. We demonstrate the prevalence of high-order irreversibility in the hyperactive regime of a biophysical model of brain dynamics, showing that our framework is both theoretically principled and empirically useful. This work challenges the view of the arrow of time as a monolithic entity, enhancing both our theoretical understanding of irreversibility and our ability to detect it in practical applications.
△ Less
Submitted 10 August, 2023;
originally announced August 2023.
-
Quantifying synergy and redundancy in multiplex networks
Authors:
Andrea I. Luppi,
Eckehard Olbrich,
Conor Finn,
Laura E. Suárez,
Fernando E. Rosas,
Pedro A. M. Mediano,
Jürgen Jost
Abstract:
Understanding how different networks relate to each other is key for obtaining a greater insight into complex systems. Here, we introduce an intuitive yet powerful framework to characterise the relationship between two networks comprising the same nodes. We showcase our framework by decomposing the shortest paths between nodes as being contributed uniquely by one or the other source network, or re…
▽ More
Understanding how different networks relate to each other is key for obtaining a greater insight into complex systems. Here, we introduce an intuitive yet powerful framework to characterise the relationship between two networks comprising the same nodes. We showcase our framework by decomposing the shortest paths between nodes as being contributed uniquely by one or the other source network, or redundantly by either, or synergistically by the two together. Our approach takes into account the networks' full topology, and it also provides insights at multiple levels of resolution: from global statistics, to individual paths of different length. We show that this approach is widely applicable, from brains to the London public transport system. In humans and across 123 other mammalian species, we demonstrate that reliance on unique contributions by long-range white matter fibers is a conserved feature of mammalian structural brain networks. Across species, we also find that efficient communication relies on significantly greater synergy between long-range and short-range fibers than expected by chance, and significantly less redundancy. Our framework may find applications to help decide how to trade-off different desiderata when designing network systems, or to evaluate their relative presence in existing systems, whether biological or artificial.
△ Less
Submitted 8 August, 2023; v1 submitted 2 June, 2023;
originally announced June 2023.
-
Dynamical noise can enhance high-order statistical structure in complex systems
Authors:
Patricio Orio,
Pedro A. M. Mediano,
Fernando E. Rosas
Abstract:
Recent research has provided a wealth of evidence highlighting the pivotal role of high-order interdependencies in supporting the information-processing capabilities of distributed complex systems. These findings may suggest that high-order interdependencies constitute a powerful resource that is, however, challenging to harness and can be readily disrupted. In this paper we contest this perspecti…
▽ More
Recent research has provided a wealth of evidence highlighting the pivotal role of high-order interdependencies in supporting the information-processing capabilities of distributed complex systems. These findings may suggest that high-order interdependencies constitute a powerful resource that is, however, challenging to harness and can be readily disrupted. In this paper we contest this perspective by demonstrating that high-order interdependencies can not only exhibit robustness to stochastic perturbations, but can in fact be enhanced by them. Using elementary cellular automata as a general testbed, our results unveil the capacity of dynamical noise to enhance the statistical regularities between agents and, intriguingly, even alter the prevailing character of their interdependencies. Furthermore, our results show that these effects are related to the high-order structure of the local rules, which affect the system's susceptibility to noise and characteristic times-scales. These results deepen our understanding of how high-order interdependencies may spontaneously emerge within distributed systems interacting with stochastic environments, thus providing an initial step towards elucidating their origin and function in complex systems like the human brain.
△ Less
Submitted 22 May, 2023;
originally announced May 2023.
-
Synergistic information supports modality integration and flexible learning in neural networks solving multiple tasks
Authors:
Alexandra M. Proca,
Fernando E. Rosas,
Andrea I. Luppi,
Daniel Bor,
Matthew Crosby,
Pedro A. M. Mediano
Abstract:
Striking progress has recently been made in understanding human cognition by analyzing how its neuronal underpinnings are engaged in different modes of information processing. Specifically, neural information can be decomposed into synergistic, redundant, and unique features, with synergistic components being particularly aligned with complex cognition. However, two fundamental questions remain un…
▽ More
Striking progress has recently been made in understanding human cognition by analyzing how its neuronal underpinnings are engaged in different modes of information processing. Specifically, neural information can be decomposed into synergistic, redundant, and unique features, with synergistic components being particularly aligned with complex cognition. However, two fundamental questions remain unanswered: (a) precisely how and why a cognitive system can become highly synergistic; and (b) how these informational states map onto artificial neural networks in various learning modes. To address these questions, here we employ an information-decomposition framework to investigate the information processing strategies adopted by simple artificial neural networks performing a variety of cognitive tasks in both supervised and reinforcement learning settings. Our results show that synergy increases as neural networks learn multiple diverse tasks. Furthermore, performance in tasks requiring integration of multiple information sources critically relies on synergistic neurons. Finally, randomly turning off neurons during training through dropout increases network redundancy, corresponding to an increase in robustness. Overall, our results suggest that while redundant information is required for robustness to perturbations in the learning process, synergistic information is used to combine information from multiple modalities -- and more generally for flexible and efficient learning. These findings open the door to new ways of investigating how and why learning systems employ specific information-processing strategies, and support the principle that the capacity for general-purpose learning critically relies in the system's information dynamics.
△ Less
Submitted 6 October, 2022;
originally announced October 2022.
-
Gradients of O-information: low-order descriptors of high-order dependencies
Authors:
Tomas Scagliarini,
Davide Nuzzi,
Yuri Antonacci,
Luca Faes,
Fernando E. Rosas,
Daniele Marinazzo,
Sebastiano Stramaglia
Abstract:
O-information is an information-theoretic metric that captures the overall balance between redundant and synergistic information shared by groups of three or more variables. To complement the global assessment provided by this metric, here we propose the gradients of the O-information as low-order descriptors that can characterise how high-order effects are localised across a system of interest. W…
▽ More
O-information is an information-theoretic metric that captures the overall balance between redundant and synergistic information shared by groups of three or more variables. To complement the global assessment provided by this metric, here we propose the gradients of the O-information as low-order descriptors that can characterise how high-order effects are localised across a system of interest. We illustrate the capabilities of the proposed framework by revealing the role of specific spins in Ising models with frustration, and on practical data analysis on US macroeconomic data. Our theoretical and empirical analyses demonstrate the potential of these gradients to highlight the contribution of variables in forming high-order informational circuits
△ Less
Submitted 1 July, 2022;
originally announced July 2022.
-
Disentangling high-order mechanisms and high-order behaviours in complex systems
Authors:
Fernando E. Rosas,
Pedro A. M. Mediano,
Andrea I. Luppi,
Thomas F. Varley,
Joseph T. Lizier,
Sebastiano Stramaglia,
Henrik J. Jensen,
Daniele Marinazzo
Abstract:
Battiston et al. (arXiv:2110.06023) provide a comprehensive overview of how investigations of complex systems should take into account interactions between more than two elements, which can be modelled by hypergraphs and studied via topological data analysis. Following a separate line of enquiry, a broad literature has developed information-theoretic tools to characterize high-order interdependenc…
▽ More
Battiston et al. (arXiv:2110.06023) provide a comprehensive overview of how investigations of complex systems should take into account interactions between more than two elements, which can be modelled by hypergraphs and studied via topological data analysis. Following a separate line of enquiry, a broad literature has developed information-theoretic tools to characterize high-order interdependencies from observed data. While these could seem to be competing approaches aiming to address the same question, in this correspondence we clarify that this is not the case, and that a complete account of higher-order phenomena needs to embrace both.
△ Less
Submitted 21 March, 2022;
originally announced March 2022.
-
Learning, compression, and leakage: Minimising classification error via meta-universal compression principles
Authors:
Fernando E. Rosas,
Pedro A. M. Mediano,
Michael Gastpar
Abstract:
Learning and compression are driven by the common aim of identifying and exploiting statistical regularities in data, which opens the door for fertile collaboration between these areas. A promising group of compression techniques for learning scenarios is normalised maximum likelihood (NML) coding, which provides strong guarantees for compression of small datasets - in contrast with more popular e…
▽ More
Learning and compression are driven by the common aim of identifying and exploiting statistical regularities in data, which opens the door for fertile collaboration between these areas. A promising group of compression techniques for learning scenarios is normalised maximum likelihood (NML) coding, which provides strong guarantees for compression of small datasets - in contrast with more popular estimators whose guarantees hold only in the asymptotic limit. Here we consider a NML-based decision strategy for supervised classification problems, and show that it attains heuristic PAC learning when applied to a wide variety of models. Furthermore, we show that the misclassification rate of our method is upper bounded by the maximal leakage, a recently proposed metric to quantify the potential of data leakage in privacy-sensitive scenarios.
△ Less
Submitted 31 January, 2021; v1 submitted 14 October, 2020;
originally announced October 2020.
-
Hyperharmonic analysis for the study of high-order information-theoretic signals
Authors:
Anibal M. Medina-Mardones,
Fernando E. Rosas,
Sebastián E. Rodríguez,
Rodrigo Cofré
Abstract:
Network representations often cannot fully account for the structural richness of complex systems spanning multiple levels of organisation. Recently proposed high-order information-theoretic signals are well-suited to capture synergistic phenomena that transcend pairwise interactions; however, the exponential-growth of their cardinality severely hinders their applicability. In this work, we combin…
▽ More
Network representations often cannot fully account for the structural richness of complex systems spanning multiple levels of organisation. Recently proposed high-order information-theoretic signals are well-suited to capture synergistic phenomena that transcend pairwise interactions; however, the exponential-growth of their cardinality severely hinders their applicability. In this work, we combine methods from harmonic analysis and combinatorial topology to construct efficient representations of high-order information-theoretic signals. The core of our method is the diagonalisation of a discrete version of the Laplace-de Rham operator, that geometrically encodes structural properties of the system. We capitalise on these ideas by developing a complete workflow for the construction of hyperharmonic representations of high-order signals, which is applicable to a wide range of scenarios.
△ Less
Submitted 22 February, 2021; v1 submitted 2 October, 2020;
originally announced October 2020.
-
Causal blankets: Theory and algorithmic framework
Authors:
Fernando E. Rosas,
Pedro A. M. Mediano,
Martin Biehl,
Shamil Chandaria,
Daniel Polani
Abstract:
We introduce a novel framework to identify perception-action loops (PALOs) directly from data based on the principles of computational mechanics. Our approach is based on the notion of causal blanket, which captures sensory and active variables as dynamical sufficient statistics -- i.e. as the "differences that make a difference." Moreover, our theory provides a broadly applicable procedure to con…
▽ More
We introduce a novel framework to identify perception-action loops (PALOs) directly from data based on the principles of computational mechanics. Our approach is based on the notion of causal blanket, which captures sensory and active variables as dynamical sufficient statistics -- i.e. as the "differences that make a difference." Moreover, our theory provides a broadly applicable procedure to construct PALOs that requires neither a steady-state nor Markovian dynamics. Using our theory, we show that every bipartite stochastic process has a causal blanket, but the extent to which this leads to an effective PALO formulation varies depending on the integrated information of the bipartition.
△ Less
Submitted 29 September, 2020; v1 submitted 28 August, 2020;
originally announced August 2020.
-
Data Disclosure under Perfect Sample Privacy
Authors:
Borzoo Rassouli,
Fernando E. Rosas,
Deniz Gunduz
Abstract:
Perfect data privacy seems to be in fundamental opposition to the economical and scientific opportunities associated with extensive data exchange. Defying this intuition, this paper develops a framework that allows the disclosure of collective properties of datasets without compromising the privacy of individual data samples. We present an algorithm to build an optimal disclosure strategy/mapping,…
▽ More
Perfect data privacy seems to be in fundamental opposition to the economical and scientific opportunities associated with extensive data exchange. Defying this intuition, this paper develops a framework that allows the disclosure of collective properties of datasets without compromising the privacy of individual data samples. We present an algorithm to build an optimal disclosure strategy/mapping, and discuss it fundamental limits on finite and asymptotically large datasets. Furthermore, we present explicit expressions to the asymptotic performance of this scheme in some scenarios, and study cases where our approach attains maximal efficiency. We finally discuss suboptimal schemes to provide sample privacy guarantees to large datasets with a reduced computational cost.
△ Less
Submitted 2 April, 2019;
originally announced April 2019.
