Showing 1–13 of 13 results for author: Lee, E D

Synthesis of innovation and obsolescence

Authors: Edward D. Lee, Christopher P. Kempes, Manfred D. Laubichler, Marcus J. Hamilton, Jeffrey W. Lockhart, Frank Neffke, Hyejin Youn, José Ignacio Arroyo, Vito D. P. Servedio, Dashun Wang, Jessika Trancik, James Evans, Vicky Chuqiao Yang, Veronica R. Cappelli, Ernesto Ortega, Yian Yin, Geoffrey B. West

Abstract: Innovation and obsolescence describe the dynamics of ever-churning social and biological systems, from the development of economic markets to scientific and technological progress to biological evolution. They have been widely discussed, but in isolation, leading to fragmented modeling of their dynamics. This poses a problem for connecting and building on what we know about their shared mechanisms… ▽ More Innovation and obsolescence describe the dynamics of ever-churning social and biological systems, from the development of economic markets to scientific and technological progress to biological evolution. They have been widely discussed, but in isolation, leading to fragmented modeling of their dynamics. This poses a problem for connecting and building on what we know about their shared mechanisms. Here we collectively propose a conceptual and mathematical framework to transcend field boundaries and to explore unifying theoretical frameworks and open challenges. We ring an optimistic note for weaving together disparate threads with key ideas from the wide and largely disconnected literature by focusing on the duality of innovation and obsolescence and by proposing a mathematical framework to unify the metaphors between constitutive elements. △ Less

Submitted 8 May, 2025; originally announced May 2025.

Common indicators hurt armed conflict prediction

Authors: Niraj Kushwaha, Woi Sok Oh, Shlok Shah, Edward D. Lee

Abstract: Are big conflicts different from small or medium size conflicts? To answer this question, we leverage fine-grained conflict data, which we map to climate, geography, infrastructure, economics, raw demographics, and demographic composition in Africa. With an unsupervised learning model, we find three overarching conflict types representing ``major unrest,'' ``local conflict,'' and ``sporadic and sp… ▽ More Are big conflicts different from small or medium size conflicts? To answer this question, we leverage fine-grained conflict data, which we map to climate, geography, infrastructure, economics, raw demographics, and demographic composition in Africa. With an unsupervised learning model, we find three overarching conflict types representing ``major unrest,'' ``local conflict,'' and ``sporadic and spillover events.'' Major unrest predominantly propagates around densely populated areas with well-developed infrastructure and flat, riparian geography. Local conflicts are in regions of median population density, are diverse socio-economically and geographically, and are often confined within country borders. Finally, sporadic and spillover conflicts remain small, often in low population density areas, with little infrastructure and poor economic conditions. The three types stratify into a hierarchy of factors that highlights population, infrastructure, economics, and geography, respectively, as the most discriminative indicators. Specifying conflict type negatively impacts the predictability of conflict intensity such as fatalities, conflict duration, and other measures of conflict size. The competitive effect is a general consequence of weak statistical dependence. Hence, we develop an empirical and bottom-up methodology to identify conflict types, knowledge of which can hurt predictability and cautions us about the limited utility of commonly available indicators. △ Less

Submitted 28 February, 2025; originally announced March 2025.

Closely estimating the entropy of sparse graph models

Authors: Edward D. Lee

Abstract: We introduce an algorithm for estimating the entropy of pairwise, probabilistic graph models by leveraging bridges between social communities and an accurate entropy estimator on sparse samples. We propose using a measure of investment from the sociological literature, Burt's structural constraint, as a heuristic for identifying bridges that partition a graph into conditionally independent compone… ▽ More We introduce an algorithm for estimating the entropy of pairwise, probabilistic graph models by leveraging bridges between social communities and an accurate entropy estimator on sparse samples. We propose using a measure of investment from the sociological literature, Burt's structural constraint, as a heuristic for identifying bridges that partition a graph into conditionally independent components. We combine this heuristic with the Nemenman-Shafee-Bialek entropy estimator to obtain a faster and more accurate estimator. We demonstrate it on the pairwise maximum entropy, or Ising, models of judicial voting, to improve naïve entropy estimates. We use our algorithm to estimate the partition function closely, which we then apply to the problem of model selection, where estimating the likelihood is difficult. This serves as an improvement over existing methods that rely on point correlation functions to test fit can be extended to other graph models with a straightforward modification of the open-source implementation. △ Less

Submitted 11 January, 2023; originally announced January 2023.

Discovering the mesoscale for chains of conflict

Authors: Niraj Kushwaha, Edward D. Lee

Abstract: Conflicts, like many social processes, are related events that span multiple scales in time, from the instantaneous to multi-year developments, and in space, from one neighborhood to continents. Yet, there is little systematic work on connecting the multiple scales, formal treatment of causality between events, and measures of uncertainty for how events are related to one another. We develop a met… ▽ More Conflicts, like many social processes, are related events that span multiple scales in time, from the instantaneous to multi-year developments, and in space, from one neighborhood to continents. Yet, there is little systematic work on connecting the multiple scales, formal treatment of causality between events, and measures of uncertainty for how events are related to one another. We develop a method for extracting related chains of events that addresses these limitations with armed conflict. Our method explicitly accounts for an adjustable spatial and temporal scale of interaction for clustering individual events from a detailed data set, the Armed Conflict Event & Location Data Project. With it, we discover a mesoscale ranging from a week to a few months and from tens to a few hundred kilometers, where long-range correlations and nontrivial dynamics relating conflict events emerge. Importantly, clusters in the mesoscale, while extracted only from conflict statistics, are identifiable with causal mechanism cited in field studies. We leverage our technique to identify zones of causal interaction around conflict hotspots that naturally incorporate uncertainties. Thus, we show how a systematic, data-driven procedure extracts social objects for study, providing a scope for scrutinizing and predicting conflict amongst other processes. △ Less

Submitted 5 December, 2022; originally announced December 2022.

Comments: 17 pages

Information consumption and size in firms

Authors: Edward D. Lee, Alan P. Kwan, Rudolf Hanel, Anjali Bhatt, Frank Neffke

Abstract: Social and biological collectives need to exchange information to persist and to function. This happens across internal networks, whose structure represents static channels through which information flows. Less studied is the quantity and variety of information transmitted. We characterize a part of the information flow, the information going into organizations, primarily business firms. We measur… ▽ More Social and biological collectives need to exchange information to persist and to function. This happens across internal networks, whose structure represents static channels through which information flows. Less studied is the quantity and variety of information transmitted. We characterize a part of the information flow, the information going into organizations, primarily business firms. We measure what firms read using a data set of hundreds of millions of records of news articles accessed by employees across millions of firms. We measure and relate quantitatively three essential aspects: reading volume, reading variety, and firm size. First we compare volume with firm size, showing that firms grow sublinearly with the volume of their reading. The scaling means that inequality in information volume exaggerates the classic Zipf's law inequality in firm size, pointing to an economy of scale in information consumption. Then, by connecting variety and volume, we show that the firms vary in their reading habits to a limited degree. Firms above a certain size become repetitive readers, consistent with the sudden onset of a coordination cost between teams, not individual employees. Finally, we relate information variety to size to show that large firms tend to increase investments in existing areas of interest instead of divesting from them to move to new areas. We argue that this reflects structural constraints in growth. The results indicate how information consumption reflects the role of internal structure, beyond individual employees, analogous to information processing in other social and biological systems. △ Less

Submitted 17 December, 2023; v1 submitted 13 October, 2022; originally announced October 2022.

Outsourcing Memory Through Niche Construction

Authors: Edward D. Lee, Jessica C. Flack, David C. Krakauer

Abstract: Adaptation to changing environments is a universal feature of life and can involve the organism modifying itself in response to the environment as well as actively modifying the environment to control selection pressures. The latter case couples the organism to environment. Then, how quickly should the organism change in response to the environment? We formulate this question in terms of how memor… ▽ More Adaptation to changing environments is a universal feature of life and can involve the organism modifying itself in response to the environment as well as actively modifying the environment to control selection pressures. The latter case couples the organism to environment. Then, how quickly should the organism change in response to the environment? We formulate this question in terms of how memory duration scales with environmental rate of change when there are trade-offs in remembering vs. forgetting. We derive a universal scaling law for optimal memory duration, taking into account memory precision as well as two components of environmental volatility, bias and stability. We find sublinear scaling with any amount of environmental volatility. We use a memory complexity measure to explore the strategic conditions (game dynamics) favoring actively reducing environmental volatility -- outsourcing memory through niche construction -- over investing in neural tissue. We predict stabilizing niche construction will evolve when neural tissue is costly, the environment is variable, and it is beneficial to be able to encode a rich repertoire of environmental states. △ Less

Submitted 7 January, 2023; v1 submitted 1 September, 2022; originally announced September 2022.

Idea engines: Unifying innovation and obsolescence from markets and genetic evolution to science

Authors: Edward D. Lee, Christopher P. Kempes, Geoffrey B. West

Abstract: Innovation and obsolescence describe dynamics of ever-churning and adapting social and biological systems, concepts that encompass field-specific formulations. We formalize the connection with a reduced model of the dynamics of the "space of the possible" (e.g. technologies, mutations, theories) to which agents (e.g. firms, organisms, scientists) couple as they grow, die, and replicate. We predict… ▽ More Innovation and obsolescence describe dynamics of ever-churning and adapting social and biological systems, concepts that encompass field-specific formulations. We formalize the connection with a reduced model of the dynamics of the "space of the possible" (e.g. technologies, mutations, theories) to which agents (e.g. firms, organisms, scientists) couple as they grow, die, and replicate. We predict three regimes: the space is finite, ever growing, or a Schumpeterian dystopia in which obsolescence drives the system to collapse. We reveal a critical boundary at which the space of the possible fluctuates dramatically in size, displaying recurrent periods of minimal and of veritable diversity. When the space is finite, corresponding to physically realizable systems, we find surprising structure. This structure predicts a taxonomy for the density of agents near and away from the innovative frontier that we compare with distributions of firm productivity, covid diversity, and citation rates for scientific publications. Remarkably, our minimal model derived from first principles aligns with empirical examples, implying a follow-the-leader dynamic in firm cost efficiency and biological evolution, whereas scientific progress reflects consensus that waits on old ideas to go obsolete. Our theory introduces a fresh and empirically testable framework for unifying innovation and obsolescence across fields. △ Less

Submitted 6 December, 2022; v1 submitted 28 March, 2022; originally announced March 2022.

A scaling theory of armed conflict avalanches

Authors: Edward D. Lee, Bryan C. Daniels, Christopher R. Myers, David C. Krakauer, Jessica C. Flack

Abstract: Armed conflict data display scaling and universal dynamics in both social and physical properties like fatalities and geographic extent. We propose a randomly branching, armed-conflict model that relates multiple properties to one another in a way consistent with data. The model incorporates a fractal lattice on which conflict spreads, uniform dynamics driving conflict growth, and regional virulen… ▽ More Armed conflict data display scaling and universal dynamics in both social and physical properties like fatalities and geographic extent. We propose a randomly branching, armed-conflict model that relates multiple properties to one another in a way consistent with data. The model incorporates a fractal lattice on which conflict spreads, uniform dynamics driving conflict growth, and regional virulence that modulates local conflict intensity. The quantitative constraints on scaling and universal dynamics we use to develop our minimal model serve more generally as a set of constraints for other models for armed conflict dynamics. We show how this approach akin to thermodynamics imparts mechanistic intuition and unifies multiple conflict properties, giving insight into causation, prediction, and intervention timing. △ Less

Submitted 29 April, 2020; originally announced April 2020.

Journal ref: Phys. Rev. E 102, 042312 (2020)

Sensitivity of collective outcomes identifies pivotal components

Authors: Edward D. Lee, Daniel M. Katz, Michael J. Bommarito II, Paul Ginsparg

Abstract: A social system is susceptible to perturbation when its collective properties depend sensitively on a few pivotal components. Using the information geometry of minimal models from statistical physics, we develop an approach to identify pivotal components to which coarse-grained, or aggregate, properties are sensitive. As an example, we introduce our approach on a reduced toy model with a median vo… ▽ More A social system is susceptible to perturbation when its collective properties depend sensitively on a few pivotal components. Using the information geometry of minimal models from statistical physics, we develop an approach to identify pivotal components to which coarse-grained, or aggregate, properties are sensitive. As an example, we introduce our approach on a reduced toy model with a median voter who always votes in the majority. The sensitivity of majority-minority divisions to changing voter behaviour pinpoints the unique role of the median. More generally, the sensitivity identifies pivotal components that precisely determine collective outcomes generated by a complex network of interactions. Using perturbations to target pivotal components in the models, we analyse datasets from political voting, finance and Twitter. Across these systems, we find remarkable variety, from systems dominated by a median-like component to those whose components behave more equally. In the context of political institutions such as courts or legislatures, our methodology can help describe how changes in voters map to new collective voting outcomes. For economic indices, differing system response reflects varying fiscal conditions across time. Thus, our information-geometric approach provides a principled, quantitative framework that may help assess the robustness of collective outcomes to targeted perturbation and compare social institutions, or even biological networks, with one another and across time. △ Less

Submitted 2 July, 2020; v1 submitted 22 September, 2019; originally announced September 2019.

Emergent regularities and scaling in armed conflict data

Authors: Edward D. Lee, Bryan C. Daniels, Christopher R. Myers, David C. Krakauer, Jessica C. Flack

Abstract: Armed conflict exhibits regularities beyond known power law distributions of fatalities and duration over varying culture and geography. We systematically cluster conflict reports from a database of $10^5$ events from Africa spanning 20 years into conflict avalanches. Conflict profiles collapse over a range of scales. Duration, diameter, extent, fatalities, and report totals satisfy mutually consi… ▽ More Armed conflict exhibits regularities beyond known power law distributions of fatalities and duration over varying culture and geography. We systematically cluster conflict reports from a database of $10^5$ events from Africa spanning 20 years into conflict avalanches. Conflict profiles collapse over a range of scales. Duration, diameter, extent, fatalities, and report totals satisfy mutually consistent scaling relations captured with a model combining geographic spread and local conflict-site growth. The emergence of such social scaling laws hints at principles guiding conflict evolution. △ Less

Submitted 29 April, 2020; v1 submitted 18 March, 2019; originally announced March 2019.

Convenient Interface to Inverse Ising (ConIII): A Python 3 Package for Solving Ising-Type Maximum Entropy Models

Authors: Edward D. Lee, Bryan C Daniels

Abstract: ConIII (pronounced CON-ee) is an open-source Python project providing a simple interface to solving the pairwise and higher order Ising model and a base for extension to other maximum entropy models. We describe the maximum entropy problem and give an overview of the algorithms that are implemented as part of ConIII (https://github.com/eltrompetero/coniii) including Monte Carlo histogram, pseudoli… ▽ More ConIII (pronounced CON-ee) is an open-source Python project providing a simple interface to solving the pairwise and higher order Ising model and a base for extension to other maximum entropy models. We describe the maximum entropy problem and give an overview of the algorithms that are implemented as part of ConIII (https://github.com/eltrompetero/coniii) including Monte Carlo histogram, pseudolikelihood, minimum probability flow, a regularized mean field method, and a cluster expansion method. Our goal is to make a variety of maximum entropy techniques accessible to those unfamiliar with the techniques and accelerate workflow for users. △ Less

Submitted 10 March, 2019; v1 submitted 24 January, 2018; originally announced January 2018.

Strong consensus on US Supreme Court spans a century

Authors: Edward D. Lee

Abstract: The US Supreme Court throughout the 20th century has been characterized as being divided between liberals and conservatives, suggesting that justices with similar ideologies would have voted similarly had they overlapped in tenure. What if they had? I build an empirical, quantitative model of this counterfactual hypothesis using pairwise maximum entropy. I infer how 36 justices from 1946-2016 woul… ▽ More The US Supreme Court throughout the 20th century has been characterized as being divided between liberals and conservatives, suggesting that justices with similar ideologies would have voted similarly had they overlapped in tenure. What if they had? I build an empirical, quantitative model of this counterfactual hypothesis using pairwise maximum entropy. I infer how 36 justices from 1946-2016 would have all voted on a Super Supreme Court. The model is strikingly consistent with a standard voting model from political science despite using $10^5$ less parameters and fitting the observed statistics better. As with historical courts, the Super Court is dominated by consensus. The rate at which consensus decays as more justices are included is extremely slow, nearly 100 years, and indicates that the modern Supreme Court is an extremely stable institution. Beyond consensus, I discover a rich structure of dissenting blocs that are distributed along a heavy-tailed Zipf's law. The heavy tail means that dominant dissenting modes fail to capture the entire spectrum of dissent. Thus, I find that Supreme Court voting over time is not low-dimensional despite implications to the contrary in historical analysis of Supreme Court voting. Although it has been long presumed that strong higher order correlations are induced by features of the cases, the institution, and the justices, I show that such complexity can be expressed in a minimal model relying only on pairwise correlations. From the perspective of model selection, this minimal model may generalize better and thus be useful for prediction of Supreme Court voting over time. △ Less

Submitted 12 April, 2018; v1 submitted 27 December, 2017; originally announced December 2017.

Statistical mechanics of the US Supreme Court

Authors: Edward D. Lee, Chase P. Broedersz, William Bialek

Abstract: We build simple models for the distribution of voting patterns in a group, using the Supreme Court of the United States as an example. The least structured, or maximum entropy, model that is consistent with the observed pairwise correlations among justices' votes is equivalent to an Ising spin glass. While all correlations (perhaps surprisingly) are positive, the effective pairwise interactions in… ▽ More We build simple models for the distribution of voting patterns in a group, using the Supreme Court of the United States as an example. The least structured, or maximum entropy, model that is consistent with the observed pairwise correlations among justices' votes is equivalent to an Ising spin glass. While all correlations (perhaps surprisingly) are positive, the effective pairwise interactions in the spin glass model have both signs, recovering some of our intuition that justices on opposite sides of the ideological spectrum should have a negative influence on one another. Despite the competing interactions, a strong tendency toward unanimity emerges from the model, and this agrees quantitatively with the data. The model shows that voting patterns are organized in a relatively simple "energy landscape," correctly predicts the extent to which each justice is correlated with the majority, and gives us a measure of the influence that justices exert on one another. These results suggest that simple models, grounded in statistical physics, can capture essential features of collective decision making quantitatively, even in a complex political context. △ Less

Submitted 20 June, 2013; originally announced June 2013.

Journal ref: Journal of Statistical Physics, Volume 160, Issue 2, pp 275–301