-
Surveying the space of descriptions of a composite system with machine learning
Authors:
Kieran A. Murphy,
Yujing Zhang,
Dani S. Bassett
Abstract:
Multivariate information theory provides a general and principled framework for understanding how the components of a complex system are connected. Existing analyses are coarse in nature -- built up from characterizations of discrete subsystems -- and can be computationally prohibitive. In this work, we propose to study the continuous space of possible descriptions of a composite system as a windo…
▽ More
Multivariate information theory provides a general and principled framework for understanding how the components of a complex system are connected. Existing analyses are coarse in nature -- built up from characterizations of discrete subsystems -- and can be computationally prohibitive. In this work, we propose to study the continuous space of possible descriptions of a composite system as a window into its organizational structure. A description consists of specific information conveyed about each of the components, and the space of possible descriptions is equivalent to the space of lossy compression schemes of the components. We introduce a machine learning framework to optimize descriptions that extremize key information theoretic quantities used to characterize organization, such as total correlation and O-information. Through case studies on spin systems, sudoku boards, and letter sequences from natural language, we identify extremal descriptions that reveal how system-wide variation emerges from individual components. By integrating machine learning into a fine-grained information theoretic analysis of composite random variables, our framework opens a new avenues for probing the structure of real-world complex systems.
△ Less
Submitted 29 May, 2025; v1 submitted 27 November, 2024;
originally announced November 2024.
-
Which bits went where? Past and future transfer entropy decomposition with the information bottleneck
Authors:
Kieran A. Murphy,
Zhuowen Yin,
Dani S. Bassett
Abstract:
Whether the system under study is a shoal of fish, a collection of neurons, or a set of interacting atmospheric and oceanic processes, transfer entropy measures the flow of information between time series and can detect possible causal relationships. Much like mutual information, transfer entropy is generally reported as a single value summarizing an amount of shared variation, yet a more fine-gra…
▽ More
Whether the system under study is a shoal of fish, a collection of neurons, or a set of interacting atmospheric and oceanic processes, transfer entropy measures the flow of information between time series and can detect possible causal relationships. Much like mutual information, transfer entropy is generally reported as a single value summarizing an amount of shared variation, yet a more fine-grained accounting might illuminate much about the processes under study. Here we propose to decompose transfer entropy and localize the bits of variation on both sides of information flow: that of the originating process's past and that of the receiving process's future. We employ the information bottleneck (IB) to compress the time series and identify the transferred entropy. We apply our method to decompose the transfer entropy in several synthetic recurrent processes and an experimental mouse dataset of concurrent behavioral and neural activity. Our approach highlights the nuanced dynamics within information flow, laying a foundation for future explorations into the intricate interplay of temporal processes in complex systems.
△ Less
Submitted 7 November, 2024;
originally announced November 2024.
-
Comparing the information content of probabilistic representation spaces
Authors:
Kieran A. Murphy,
Sam Dillavou,
Dani S. Bassett
Abstract:
Probabilistic representation spaces convey information about a dataset and are shaped by factors such as the training data, network architecture, and loss function. Comparing the information content of such spaces is crucial for understanding the learning process, yet most existing methods assume point-based representations, neglecting the distributional nature of probabilistic spaces. To address…
▽ More
Probabilistic representation spaces convey information about a dataset and are shaped by factors such as the training data, network architecture, and loss function. Comparing the information content of such spaces is crucial for understanding the learning process, yet most existing methods assume point-based representations, neglecting the distributional nature of probabilistic spaces. To address this gap, we propose two information-theoretic measures to compare general probabilistic representation spaces by extending classic methods to compare the information content of hard clustering assignments. Additionally, we introduce a lightweight method of estimation that is based on fingerprinting a representation space with a sample of the dataset, designed for scenarios where the communicated information is limited to a few bits. We demonstrate the utility of these measures in three case studies. First, in the context of unsupervised disentanglement, we identify recurring information fragments within individual latent dimensions of VAE and InfoGAN ensembles. Second, we compare the full latent spaces of models and reveal consistent information content across datasets and methods, despite variability during training. Finally, we leverage the differentiability of our measures to perform model fusion, synthesizing the information content of weak learners into a single, coherent representation. Across these applications, the direct comparison of information content offers a natural basis for characterizing the processing of information.
△ Less
Submitted 18 February, 2025; v1 submitted 31 May, 2024;
originally announced May 2024.
-
Mechanical prions: Self-assembling microstructures
Authors:
Mathieu Ouellet,
Dani S. Bassett,
Lee C. Bassett,
Kieran A. Murphy,
Shubhankar P. Patankar
Abstract:
Prions are misfolded proteins that transmit their structural arrangement to neighboring proteins. In biological systems, prion dynamics can produce a variety of complex functional outcomes. Yet, an understanding of prionic causes has been hampered by the fact that few computational models exist that allow for experimental design, hypothesis testing, and control. Here, we identify essential prionic…
▽ More
Prions are misfolded proteins that transmit their structural arrangement to neighboring proteins. In biological systems, prion dynamics can produce a variety of complex functional outcomes. Yet, an understanding of prionic causes has been hampered by the fact that few computational models exist that allow for experimental design, hypothesis testing, and control. Here, we identify essential prionic properties and present a biologically inspired model of prions using simple mechanical structures capable of undergoing complex conformational change. We demonstrate the utility of our approach by designing a prototypical mechanical prion and validating its properties experimentally. Our work provides a design framework for harnessing and manipulating prionic properties in natural and artificial systems.
△ Less
Submitted 11 January, 2025; v1 submitted 7 February, 2024;
originally announced February 2024.
-
Machine-learning optimized measurements of chaotic dynamical systems via the information bottleneck
Authors:
Kieran A. Murphy,
Dani S. Bassett
Abstract:
Deterministic chaos permits a precise notion of a "perfect measurement" as one that, when obtained repeatedly, captures all of the information created by the system's evolution with minimal redundancy. Finding an optimal measurement is challenging, and has generally required intimate knowledge of the dynamics in the few cases where it has been done. We establish an equivalence between a perfect me…
▽ More
Deterministic chaos permits a precise notion of a "perfect measurement" as one that, when obtained repeatedly, captures all of the information created by the system's evolution with minimal redundancy. Finding an optimal measurement is challenging, and has generally required intimate knowledge of the dynamics in the few cases where it has been done. We establish an equivalence between a perfect measurement and a variant of the information bottleneck. As a consequence, we can employ machine learning to optimize measurement processes that efficiently extract information from trajectory data. We obtain approximately optimal measurements for multiple chaotic maps and lay the necessary groundwork for efficient information extraction from general time series.
△ Less
Submitted 19 March, 2024; v1 submitted 8 November, 2023;
originally announced November 2023.
-
Intrinsically motivated graph exploration using network theories of human curiosity
Authors:
Shubhankar P. Patankar,
Mathieu Ouellet,
Juan Cervino,
Alejandro Ribeiro,
Kieran A. Murphy,
Dani S. Bassett
Abstract:
Intrinsically motivated exploration has proven useful for reinforcement learning, even without additional extrinsic rewards. When the environment is naturally represented as a graph, how to guide exploration best remains an open question. In this work, we propose a novel approach for exploring graph-structured data motivated by two theories of human curiosity: the information gap theory and the co…
▽ More
Intrinsically motivated exploration has proven useful for reinforcement learning, even without additional extrinsic rewards. When the environment is naturally represented as a graph, how to guide exploration best remains an open question. In this work, we propose a novel approach for exploring graph-structured data motivated by two theories of human curiosity: the information gap theory and the compression progress theory. The theories view curiosity as an intrinsic motivation to optimize for topological features of subgraphs induced by nodes visited in the environment. We use these proposed features as rewards for graph neural-network-based reinforcement learning. On multiple classes of synthetically generated graphs, we find that trained agents generalize to longer exploratory walks and larger environments than are seen during training. Our method computes more efficiently than the greedy evaluation of the relevant topological properties. The proposed intrinsic motivations bear particular relevance for recommender systems. We demonstrate that next-node recommendations considering curiosity are more predictive of human choices than PageRank centrality in several real-world graph environments.
△ Less
Submitted 1 December, 2023; v1 submitted 10 July, 2023;
originally announced July 2023.
-
Information decomposition in complex systems via machine learning
Authors:
Kieran A. Murphy,
Dani S. Bassett
Abstract:
One of the fundamental steps toward understanding a complex system is identifying variation at the scale of the system's components that is most relevant to behavior on a macroscopic scale. Mutual information provides a natural means of linking variation across scales of a system due to its independence of functional relationship between observables. However, characterizing the manner in which inf…
▽ More
One of the fundamental steps toward understanding a complex system is identifying variation at the scale of the system's components that is most relevant to behavior on a macroscopic scale. Mutual information provides a natural means of linking variation across scales of a system due to its independence of functional relationship between observables. However, characterizing the manner in which information is distributed across a set of observables is computationally challenging and generally infeasible beyond a handful of measurements. Here we propose a practical and general methodology that uses machine learning to decompose the information contained in a set of measurements by jointly optimizing a lossy compression of each measurement. Guided by the distributed information bottleneck as a learning objective, the information decomposition identifies the variation in the measurements of the system state most relevant to specified macroscale behavior. We focus our analysis on two paradigmatic complex systems: a Boolean circuit and an amorphous material undergoing plastic deformation. In both examples, the large amount of entropy of the system state is decomposed, bit by bit, in terms of what is most related to macroscale behavior. The identification of meaningful variation in data, with the full generality brought by information theory, is made practical for studying the connection between micro- and macroscale structure in complex systems.
△ Less
Submitted 18 March, 2024; v1 submitted 10 July, 2023;
originally announced July 2023.
-
Interpretability with full complexity by constraining feature information
Authors:
Kieran A. Murphy,
Dani S. Bassett
Abstract:
Interpretability is a pressing issue for machine learning. Common approaches to interpretable machine learning constrain interactions between features of the input, rendering the effects of those features on a model's output comprehensible but at the expense of model complexity. We approach interpretability from a new angle: constrain the information about the features without restricting the comp…
▽ More
Interpretability is a pressing issue for machine learning. Common approaches to interpretable machine learning constrain interactions between features of the input, rendering the effects of those features on a model's output comprehensible but at the expense of model complexity. We approach interpretability from a new angle: constrain the information about the features without restricting the complexity of the model. Borrowing from information theory, we use the Distributed Information Bottleneck to find optimal compressions of each feature that maximally preserve information about the output. The learned information allocation, by feature and by feature value, provides rich opportunities for interpretation, particularly in problems with many features and complex feature interactions. The central object of analysis is not a single trained model, but rather a spectrum of models serving as approximations that leverage variable amounts of information about the inputs. Information is allocated to features by their relevance to the output, thereby solving the problem of feature selection by constructing a learned continuum of feature inclusion-to-exclusion. The optimal compression of each feature -- at every stage of approximation -- allows fine-grained inspection of the distinctions among feature values that are most impactful for prediction. We develop a framework for extracting insight from the spectrum of approximate models and demonstrate its utility on a range of tabular datasets.
△ Less
Submitted 30 November, 2022;
originally announced November 2022.
-
Characterizing information loss in a chaotic double pendulum with the Information Bottleneck
Authors:
Kieran A. Murphy,
Dani S. Bassett
Abstract:
A hallmark of chaotic dynamics is the loss of information with time. Although information loss is often expressed through a connection to Lyapunov exponents -- valid in the limit of high information about the system state -- this picture misses the rich spectrum of information decay across different levels of granularity. Here we show how machine learning presents new opportunities for the study o…
▽ More
A hallmark of chaotic dynamics is the loss of information with time. Although information loss is often expressed through a connection to Lyapunov exponents -- valid in the limit of high information about the system state -- this picture misses the rich spectrum of information decay across different levels of granularity. Here we show how machine learning presents new opportunities for the study of information loss in chaotic dynamics, with a double pendulum serving as a model system. We use the Information Bottleneck as a training objective for a neural network to extract information from the state of the system that is optimally predictive of the future state after a prescribed time horizon. We then decompose the optimally predictive information by distributing a bottleneck to each state variable, recovering the relative importance of the variables in determining future evolution. The framework we develop is broadly applicable to chaotic systems and pragmatic to apply, leveraging data and machine learning to monitor the limits of predictability and map out the loss of information.
△ Less
Submitted 25 October, 2022;
originally announced October 2022.
-
The Distributed Information Bottleneck reveals the explanatory structure of complex systems
Authors:
Kieran A. Murphy,
Dani S. Bassett
Abstract:
The fruits of science are relationships made comprehensible, often by way of approximation. While deep learning is an extremely powerful way to find relationships in data, its use in science has been hindered by the difficulty of understanding the learned relationships. The Information Bottleneck (IB) is an information theoretic framework for understanding a relationship between an input and an ou…
▽ More
The fruits of science are relationships made comprehensible, often by way of approximation. While deep learning is an extremely powerful way to find relationships in data, its use in science has been hindered by the difficulty of understanding the learned relationships. The Information Bottleneck (IB) is an information theoretic framework for understanding a relationship between an input and an output in terms of a trade-off between the fidelity and complexity of approximations to the relationship. Here we show that a crucial modification -- distributing bottlenecks across multiple components of the input -- opens fundamentally new avenues for interpretable deep learning in science. The Distributed Information Bottleneck throttles the downstream complexity of interactions between the components of the input, deconstructing a relationship into meaningful approximations found through deep learning without requiring custom-made datasets or neural network architectures. Applied to a complex system, the approximations illuminate aspects of the system's nature by restricting -- and monitoring -- the information about different components incorporated into the approximation. We demonstrate the Distributed IB's explanatory utility in systems drawn from applied mathematics and condensed matter physics. In the former, we deconstruct a Boolean circuit into approximations that isolate the most informative subsets of input components without requiring exhaustive search. In the latter, we localize information about future plastic rearrangement in the static structure of a sheared glass, and find the information to be more or less diffuse depending on the system's preparation. By way of a principled scheme of approximations, the Distributed IB brings much-needed interpretability to deep learning and enables unprecedented analysis of information flow through a system.
△ Less
Submitted 15 April, 2022;
originally announced April 2022.
-
Learning ABCs: Approximate Bijective Correspondence for isolating factors of variation with weak supervision
Authors:
Kieran A. Murphy,
Varun Jampani,
Srikumar Ramalingam,
Ameesh Makadia
Abstract:
Representational learning forms the backbone of most deep learning applications, and the value of a learned representation is intimately tied to its information content regarding different factors of variation. Finding good representations depends on the nature of supervision and the learning algorithm. We propose a novel algorithm that utilizes a weak form of supervision where the data is partiti…
▽ More
Representational learning forms the backbone of most deep learning applications, and the value of a learned representation is intimately tied to its information content regarding different factors of variation. Finding good representations depends on the nature of supervision and the learning algorithm. We propose a novel algorithm that utilizes a weak form of supervision where the data is partitioned into sets according to certain inactive (common) factors of variation which are invariant across elements of each set. Our key insight is that by seeking correspondence between elements of different sets, we learn strong representations that exclude the inactive factors of variation and isolate the active factors that vary within all sets. As a consequence of focusing on the active factors, our method can leverage a mix of set-supervised and wholly unsupervised data, which can even belong to a different domain. We tackle the challenging problem of synthetic-to-real object pose transfer, without pose annotations on anything, by isolating pose information which generalizes to the category level and across the synthetic/real domain gap. The method can also boost performance in supervised settings, by strengthening intermediate representations, as well as operate in practically attainable scenarios with set-supervised natural images, where quantity is limited and nuisance factors of variation are more plentiful.
△ Less
Submitted 30 March, 2022; v1 submitted 4 March, 2021;
originally announced March 2021.
-
Memory in nonmonotonic stress relaxation of a granular system
Authors:
Kieran A. Murphy,
Jonathon W. Kruppe,
Heinrich M. Jaeger
Abstract:
We demonstrate experimentally that a granular packing of glass spheres is capable of storing memory of multiple strain states in the dynamic process of stress relaxation. Modeling the system as a non-interacting population of relaxing elements, we find that the functional form of the predicted relaxation requires a quantitative correction which grows in severity with each additional memory and is…
▽ More
We demonstrate experimentally that a granular packing of glass spheres is capable of storing memory of multiple strain states in the dynamic process of stress relaxation. Modeling the system as a non-interacting population of relaxing elements, we find that the functional form of the predicted relaxation requires a quantitative correction which grows in severity with each additional memory and is suggestive of interactions between elements. Our findings have implications for the broad class of soft matter systems that display memory and anomalous relaxation.
△ Less
Submitted 16 November, 2019; v1 submitted 4 November, 2019;
originally announced November 2019.
-
The intertwined roles of particle shape and surface roughness in controlling the shear strength of a granular material
Authors:
Kieran A. Murphy,
Arthur K. MacKeith,
Leah K. Roth,
Heinrich M. Jaeger
Abstract:
Both the shape of individual particles and their surface properties contribute to the strength of a granular material under shear. Here we show the degree to which these two aspects can be intertwined. In experiments on assemblies of 3D printed, convex lens-shaped particles, we measure the stress-strain response under repeated compressive loading and find that the aggregate's shear strength falls…
▽ More
Both the shape of individual particles and their surface properties contribute to the strength of a granular material under shear. Here we show the degree to which these two aspects can be intertwined. In experiments on assemblies of 3D printed, convex lens-shaped particles, we measure the stress-strain response under repeated compressive loading and find that the aggregate's shear strength falls rapidly when compared to other particle shapes. We probe the granular material at mm-scales with X-ray computed tomography and $μ$m-scales with high-resolution surface metrology to look for the cause of the degradation. We find that wear due to accumulated deformation smooths out the lens surfaces in a controlled and systematic manner that correlates with a significant loss of shear strength observed for the assembly as a whole. The sensitivity of lenses to changes in surface properties contrasts with results for assemblies of 3D printed tetrahedra and spheres, which under the same load cycling are found to exhibit only minor degradation in strength. This case study provides insight into the relationship between particle shape, surface wear, and the overall material response, and suggests new strategies when designing a granular material with desired evolution of properties under repeated deformation.
△ Less
Submitted 24 June, 2019; v1 submitted 8 February, 2019;
originally announced February 2019.
-
Edges control clustering in levitated granular matter
Authors:
Melody Xuan Lim,
Kieran A. Murphy,
Heinrich M. Jaeger
Abstract:
The properties of small clusters depend dramatically on the interactions between their constituent particles. However, it remains challenging to design and tune the interactions between macroscopic particles, such as in a granular material. Here, we use acoustic levitation to trap macroscopic grains and induce forces between them. Our main results show that particles levitated in an acoustic field…
▽ More
The properties of small clusters depend dramatically on the interactions between their constituent particles. However, it remains challenging to design and tune the interactions between macroscopic particles, such as in a granular material. Here, we use acoustic levitation to trap macroscopic grains and induce forces between them. Our main results show that particles levitated in an acoustic field prefer to make contact along sharp edges. The radius of curvature of the edges directly controls the magnitude of these forces. These highly directional interactions, combined with local contact forces, give rise to a diverse array of cluster shapes. Our results open up new possibilities for the design of specific forces between macroscopic particles, directing their assembly, and actuating their motion.
△ Less
Submitted 6 June, 2019; v1 submitted 24 January, 2019;
originally announced January 2019.
-
Transforming mesoscale granular plasticity through particle shape
Authors:
Kieran A. Murphy,
Karin A. Dahmen,
Heinrich M. Jaeger
Abstract:
When an amorphous material is strained beyond the point of yielding it enters a state of continual reconfiguration via dissipative, avalanche-like slip events that relieve built-up local stress. However, how the statistics of such events depend on local interactions among the constituent units remains debated. To address this we perform experiments on granular material in which we use particle sha…
▽ More
When an amorphous material is strained beyond the point of yielding it enters a state of continual reconfiguration via dissipative, avalanche-like slip events that relieve built-up local stress. However, how the statistics of such events depend on local interactions among the constituent units remains debated. To address this we perform experiments on granular material in which we use particle shape to vary the interactions systematically. Granular material, confined under constant pressure boundary conditions, is uniaxially compressed while stress is measured and internal rearrangements are imaged with x-rays. We introduce volatility, a quantity from economic theory, as a powerful new tool to quantify the magnitude of stress fluctuations, finding systematic, shape-dependent trends. For all 22 investigated shapes the magnitude $s$ of relaxation events is well-fit by a truncated power law distribution $P(s)\sim {s}^{-τ} exp(-s/s^*)$, as has been proposed within the context of plasticity models. The power law exponent $τ$ for all shapes tested clusters around $τ=$ 1.5, within experimental uncertainty covering the range 1.3 - 1.7. The shape independence of $τ$ and its compatibility with mean field models indicate that the granularity of the system, but not particle shape, modifies the stress redistribution after a slip event away from that of continuum elasticity. Meanwhile, the characteristic maximum event size $s^*$ changes by two orders of magnitude and tracks the shape dependence of volatility. Particle shape in granular materials is therefore a powerful new factor influencing the distance at which an amorphous system operates from scale-free criticality. These experimental results are not captured by current models and suggest a need to reexamine the mechanisms driving mesoscale plastic deformation in amorphous systems.
△ Less
Submitted 21 December, 2018; v1 submitted 19 August, 2018;
originally announced August 2018.
-
Freestanding loadbearing structures with Z-shaped particles
Authors:
Kieran A. Murphy,
Nikolaj Reiser,
Darius Choksy,
Clare E. Singer,
Heinrich M. Jaeger
Abstract:
Architectural structures such as masonry walls or columns exhibit a slender verticality, in contrast to the squat, sloped forms obtained with typical unconfined granular materials. Here we demonstrate the ability to create freestanding, weight-bearing, similarly slender and vertical structures by the simple pouring of suitably shaped dry particles into a mold that is subsequently removed. Combinin…
▽ More
Architectural structures such as masonry walls or columns exhibit a slender verticality, in contrast to the squat, sloped forms obtained with typical unconfined granular materials. Here we demonstrate the ability to create freestanding, weight-bearing, similarly slender and vertical structures by the simple pouring of suitably shaped dry particles into a mold that is subsequently removed. Combining experiments and simulations we explore a family of particle types that can entangle through their non-convex, hooked shape. We show that Z-shaped particles produce granular aggregates which can either be fluid and pourable, or solid and rigid enough to maintain vertical interfaces and build freestanding columns of large aspect ratio (>10) that support compressive loads without external confinement. We investigate the stability of such columns with uniaxial compression, bending, and vibration tests and compare with other particle types including U-shaped particles and rods. We find a pronounced anisotropy in the internal stress propagation together with strong strain-stiffening, which stabilizes rather than destabilizes the structures under load.
△ Less
Submitted 19 October, 2015;
originally announced October 2015.
