-
Kendall Correlation Coefficients for Portfolio Optimization
Authors:
Tomas Espana,
Victor Le Coz,
Matteo Smerlak
Abstract:
Markowitz's optimal portfolio relies on the accurate estimation of correlations between asset returns, a difficult problem when the number of observations is not much larger than the number of assets. Using powerful results from random matrix theory, several schemes have been developed to "clean" the eigenvalues of empirical correlation matrices. By contrast, the (in practice equally important) pr…
▽ More
Markowitz's optimal portfolio relies on the accurate estimation of correlations between asset returns, a difficult problem when the number of observations is not much larger than the number of assets. Using powerful results from random matrix theory, several schemes have been developed to "clean" the eigenvalues of empirical correlation matrices. By contrast, the (in practice equally important) problem of correctly estimating the eigenvectors of the correlation matrix has received comparatively little attention. Here we discuss a class of correlation estimators generalizing Kendall's rank correlation coefficient which improve the estimation of both eigenvalues and eigenvectors in data-poor regimes. Using both synthetic and real financial data, we show that these generalized correlation coefficients yield Markowitz portfolios with lower out-of-sample risk than those obtained with rotationally invariant estimators. Central to these results is a property shared by all Kendall-like estimators but not with classical correlation coefficients: zero eigenvalues only appear when the number of assets becomes proportional to the square of the number of data points.
△ Less
Submitted 22 October, 2024;
originally announced October 2024.
-
Complexity-stability relationships in competitive disordered dynamical systems
Authors:
Onofrio Mazzarisi,
Matteo Smerlak
Abstract:
Robert May famously used random matrix theory to predict that large, complex systems cannot admit stable fixed points. However, this general conclusion is not always supported by empirical observation: from cells to biomes, biological systems are large, complex, and often stable. In this paper, we revisit May's argument in light of recent developments in both ecology and random matrix theory. We f…
▽ More
Robert May famously used random matrix theory to predict that large, complex systems cannot admit stable fixed points. However, this general conclusion is not always supported by empirical observation: from cells to biomes, biological systems are large, complex, and often stable. In this paper, we revisit May's argument in light of recent developments in both ecology and random matrix theory. We focus on competitive systems, and, using a nonlinear generalization of the competitive Lotka-Volterra model, we show that there are, in fact, two kinds of complexity-stability relationships in disordered dynamical systems: if self-interactions grow faster with density than cross-interactions, complexity is destabilizing; but if cross-interactions grow faster than self-interactions, complexity is stabilizing.
△ Less
Submitted 7 November, 2024; v1 submitted 16 March, 2024;
originally announced March 2024.
-
Perturbation theory without power series: iterative construction of non-analytic operator spectra
Authors:
Matteo Smerlak
Abstract:
It is well known that quantum-mechanical perturbation theory often give rise to divergent series that require proper resummation. Here I discuss simple ways in which these divergences can be avoided in the first place. Using the elementary technique of relaxed fixed-point iteration, I obtain convergent expressions for various challenging ground states wavefunctions, including quartic, sextic and o…
▽ More
It is well known that quantum-mechanical perturbation theory often give rise to divergent series that require proper resummation. Here I discuss simple ways in which these divergences can be avoided in the first place. Using the elementary technique of relaxed fixed-point iteration, I obtain convergent expressions for various challenging ground states wavefunctions, including quartic, sextic and octic anharmonic oscillators, the hydrogenic Zeeman problem, and the Herbst-Simon Hamiltonian (with finite energy but vanishing Rayleigh-Schrödinger coefficients), all at arbitarily strong coupling. These results challenge the notion that non-analytic functions of coupling constants are intrinsically "non-perturbative". A possible application to exact diagonalization is briefly discussed.
△ Less
Submitted 27 October, 2022; v1 submitted 11 May, 2021;
originally announced May 2021.
-
Intermittency as metastability: a predictive approach to evolution in disordered environments
Authors:
Matteo Smerlak
Abstract:
Many systems across the sciences evolve through a combination of multiplicative growth and diffusive transport. In the presence of disorder, these systems tend to form localized structures which alternate between long periods of relative stasis and short bursts of activity. This behaviour, known as intermittency in physics and punctuated equilibrium in evolutionary theory, is difficult to forecast…
▽ More
Many systems across the sciences evolve through a combination of multiplicative growth and diffusive transport. In the presence of disorder, these systems tend to form localized structures which alternate between long periods of relative stasis and short bursts of activity. This behaviour, known as intermittency in physics and punctuated equilibrium in evolutionary theory, is difficult to forecast; in particular there is no general principle to locate the regions where the system will settle, how long it will stay there, or where it will jump next. Here I introduce a predictive theory of linear intermittency that closes these gaps. I show that any positive linear system can be mapped onto a generalization of the "maximal entropy random walk", a Markov process on graphs with non-local transition rates. This construction reveals the localization islands as local minima of an effective potential, and intermittent jumps as barrier crossings in that potential. My results unify the concepts of intermittency in linear systems and Markovian metastability, and provide a generally applicable method to reduce, and predict, the dynamics of disordered linear systems. Applications span physics, evolutionary dynamics and epidemiology.
△ Less
Submitted 10 September, 2020; v1 submitted 8 September, 2020;
originally announced September 2020.
-
Effective potential reveals evolutionary trajectories in complex fitness landscapes
Authors:
Matteo Smerlak
Abstract:
Growing efforts to measure fitness landscapes in molecular and microbial systems are premised on a tight relationship between landscape topography and evolutionary trajectories. This relationship, however, is far from being straightforward: depending on their mutation rate, Darwinian populations can climb the closest fitness peak (survival of the fittest), settle in lower regions with higher mutat…
▽ More
Growing efforts to measure fitness landscapes in molecular and microbial systems are premised on a tight relationship between landscape topography and evolutionary trajectories. This relationship, however, is far from being straightforward: depending on their mutation rate, Darwinian populations can climb the closest fitness peak (survival of the fittest), settle in lower regions with higher mutational robustness (survival of the flattest), or fail to adapt altogether (error catastrophes). These bifurcations highlight that evolution does not necessarily drive populations "from lower peak to higher peak", as Wright imagined. The problem therefore remains: how exactly does a complex landscape topography constrain evolution, and can we predict where it will go next? Here I introduce a generalization of quasispecies theory which identifies metastable evolutionary states as minima of an effective potential. From this representation I derive a coarse-grained, Markov state model of evolution, which in turn forms a basis for evolutionary predictions across a wide range of mutation rates. Because the effective potential is related to the ground state of a quantum Hamiltonian, my approach could stimulate fruitful interactions between evolutionary dynamics and quantum many-body theory
△ Less
Submitted 12 December, 2019;
originally announced December 2019.
-
Natural selection as coarsening
Authors:
Matteo Smerlak
Abstract:
Analogies between evolutionary dynamics and statistical mechanics, such as Fisher's second-law-like "fundamental theorem of natural selection" and Wright's "fitness landscapes", have had a deep and fruitful influence on the development of evolutionary theory. Here I discuss a new conceptual link between evolution and statistical physics. I argue that natural selection can be viewed as a coarsening…
▽ More
Analogies between evolutionary dynamics and statistical mechanics, such as Fisher's second-law-like "fundamental theorem of natural selection" and Wright's "fitness landscapes", have had a deep and fruitful influence on the development of evolutionary theory. Here I discuss a new conceptual link between evolution and statistical physics. I argue that natural selection can be viewed as a coarsening phenomenon, similar to the growth of domain size in quenched magnets or to Ostwald ripening in alloys and emulsions. In particular, I show that the most remarkable features of coarsening---scaling and self-similarity---have strict equivalents in evolutionary dynamics. This analogy has three main virtues: it brings a set of well-developed mathematical tools to bear on evolutionary dynamics; it suggests new problems in theoretical evolution; and it provides coarsening physics with a new exactly soluble model.
△ Less
Submitted 6 October, 2017; v1 submitted 17 July, 2017;
originally announced July 2017.
-
Universal statistics of selected values
Authors:
Matteo Smerlak,
Ahmed Youssef
Abstract:
Selection, the tendency of some traits to become more frequent than others in a population under the influence of some (natural or artificial) agency, is a key component of Darwinian evolution and countless other natural and social phenomena. Yet a general theory of selection, analogous to the Fisher-Tippett-Gnedenko theory of extreme events, is lacking. Here we introduce a probabilistic definitio…
▽ More
Selection, the tendency of some traits to become more frequent than others in a population under the influence of some (natural or artificial) agency, is a key component of Darwinian evolution and countless other natural and social phenomena. Yet a general theory of selection, analogous to the Fisher-Tippett-Gnedenko theory of extreme events, is lacking. Here we introduce a probabilistic definition of selection and show that selected values are attracted to a universal family of limiting distributions. The universality classes and scaling exponents are determined by the tail thickness of the random variable under selection. Our results are supported by data from molecular biology, agriculture and sport.
△ Less
Submitted 2 December, 2016;
originally announced December 2016.
-
Minimum relative entropy distributions with a large mean are Gaussian
Authors:
Matteo Smerlak
Abstract:
We consider the following frustrated optimization problem: given a prior probability distribution $q$, find the distribution $p$ minimizing the relative entropy with respect to $q$ such that $\textrm{mean}(p)$ is fixed and large. We show that solutions to this problem are asymptotically Gaussian. As an application we derive an $H$-type theorem for evolutionary dynamics: the entropy of the (standar…
▽ More
We consider the following frustrated optimization problem: given a prior probability distribution $q$, find the distribution $p$ minimizing the relative entropy with respect to $q$ such that $\textrm{mean}(p)$ is fixed and large. We show that solutions to this problem are asymptotically Gaussian. As an application we derive an $H$-type theorem for evolutionary dynamics: the entropy of the (standardized) distribution of fitness of a population evolving under natural selection is eventually increasing.
△ Less
Submitted 26 May, 2016;
originally announced May 2016.
-
Limiting fitness distributions in evolutionary dynamics
Authors:
Matteo Smerlak,
Ahmed Youssef
Abstract:
Darwinian evolution can be modeled in general terms as a flow in the space of fitness (i.e. reproductive rate) distributions. In the diffusion approximation, Tsimring et al. have showed that this flow admits "fitness wave" solutions: Gaussian-shape fitness distributions moving towards higher fitness values at constant speed. Here we show more generally that evolving fitness distributions are attra…
▽ More
Darwinian evolution can be modeled in general terms as a flow in the space of fitness (i.e. reproductive rate) distributions. In the diffusion approximation, Tsimring et al. have showed that this flow admits "fitness wave" solutions: Gaussian-shape fitness distributions moving towards higher fitness values at constant speed. Here we show more generally that evolving fitness distributions are attracted to a one-parameter family of distributions with a fixed parabolic relationship between skewness and kurtosis. Unlike fitness waves, this statistical pattern encompasses both positive and negative (a.k.a. purifying) selection and is not restricted to rapidly adapting populations. Moreover we find that the mean fitness of a population under the selection of pre-existing variation is a power-law function of time, as observed in microbiological evolution experiments but at variance with fitness wave theory. At the conceptual level, our results can be viewed as the resolution of the "dynamic insufficiency" of Fisher's fundamental theorem of natural selection. Our predictions are in good agreement with numerical simulations.
△ Less
Submitted 16 February, 2016; v1 submitted 1 November, 2015;
originally announced November 2015.
-
Thermodynamics of inequalities: from precariousness to economic stratification
Authors:
Matteo Smerlak
Abstract:
Growing economic inequalities are observed in several countries throughout the world. Following Pareto, the power-law structure of these inequalities has been the subject of much theoretical and empirical work. But their nonequilibrium dynamics, e.g. after a policy change, remains incompletely understood. Here we introduce a thermodynamical theory of inequalities based on the analogy between econo…
▽ More
Growing economic inequalities are observed in several countries throughout the world. Following Pareto, the power-law structure of these inequalities has been the subject of much theoretical and empirical work. But their nonequilibrium dynamics, e.g. after a policy change, remains incompletely understood. Here we introduce a thermodynamical theory of inequalities based on the analogy between economic stratification and statistical entropy. Within this framework we identify the combination of upward mobility with precariousness as a fundamental driver of inequality. We formalize this statement by a "second-law" inequality displaying upward mobility and precariousness as thermodynamic conjugate variables. We estimate the time scale for the "relaxation" of the wealth distribution after a sudden change of the after-tax return on capital. Our method can be generalized to gain insight into the dynamics of inequalities in any Markovian model of socioeconomic interactions.
△ Less
Submitted 30 November, 2014; v1 submitted 24 June, 2014;
originally announced June 2014.
-
Stochastic thermodynamics of entropic transport
Authors:
Matteo Smerlak
Abstract:
Seifert derived an exact fluctuation relation for diffusion processes using the concept of "stochastic system entropy". In this note we extend his formalism to entropic transport. We introduce the notion of relative stochastic entropy, or "relative surprisal", and use it to generalize Seifert's system/medium decomposition of the total entropy. This result allows to apply the concepts of stochastic…
▽ More
Seifert derived an exact fluctuation relation for diffusion processes using the concept of "stochastic system entropy". In this note we extend his formalism to entropic transport. We introduce the notion of relative stochastic entropy, or "relative surprisal", and use it to generalize Seifert's system/medium decomposition of the total entropy. This result allows to apply the concepts of stochastic thermodynamics to diffusion processes in confined geometries, such as ion channels, cellular pores or nanoporous materials. It can be seen as the equivalent for diffusion processes of Esposito and Schaller's generalized fluctuation theorem for "Maxwell demon feedbacks".
△ Less
Submitted 29 November, 2012; v1 submitted 4 July, 2012;
originally announced July 2012.
-
Universality in p-spin glasses with correlated disorder
Authors:
Valentin Bonzom,
Razvan Gurau,
Matteo Smerlak
Abstract:
We introduce a new method, based on the recently developed random tensor theory, to study the p-spin glass model with non-Gaussian, correlated disorder. Using a suitable generalization of Gurau's theorem on the universality of the large N limit of the p-unitary ensemble of random tensors, we exhibit an infinite family of such non-Gaussian distributions which leads to same low temperature phase as…
▽ More
We introduce a new method, based on the recently developed random tensor theory, to study the p-spin glass model with non-Gaussian, correlated disorder. Using a suitable generalization of Gurau's theorem on the universality of the large N limit of the p-unitary ensemble of random tensors, we exhibit an infinite family of such non-Gaussian distributions which leads to same low temperature phase as the Gaussian distribution. While this result is easy to show (and well known) for uncorrelated disorder, its robustness with respect to strong quenched correlations is surprising. We show in detail how the critical temperature is renormalized by these correlations. We close with a speculation on possible applications of random tensor theory to finite-range spin glass models.
△ Less
Submitted 9 December, 2012; v1 submitted 24 June, 2012;
originally announced June 2012.
-
Self-organization without heat: the geometric ratchet effect
Authors:
Matteo Smerlak,
Ahmed Youssef
Abstract:
We point out a surprising feature of diffusion in inhomogeneous media: under suitable conditions, the rectification of the Brownian paths by a diffusivity gradient can result in initially spread tracers spontaneously concentrating. This "geometric ratchet effect" demonstrates that, in violation of the classical statements of the second law of (non-equilibrium) thermodynamics, self-organization can…
▽ More
We point out a surprising feature of diffusion in inhomogeneous media: under suitable conditions, the rectification of the Brownian paths by a diffusivity gradient can result in initially spread tracers spontaneously concentrating. This "geometric ratchet effect" demonstrates that, in violation of the classical statements of the second law of (non-equilibrium) thermodynamics, self-organization can take place in thermodynamic systems at local equilibrium without heat being produced or exchanged with the environment. We stress the role of Bayesian priors in a suitable reformulation of the second law accommodating this geometric ratchet effect.
△ Less
Submitted 15 June, 2012;
originally announced June 2012.
-
On the inertia of heat
Authors:
Matteo Smerlak
Abstract:
Does heat have inertia? This question is at the core of a long-standing controversy on Eckart's dissipative relativistic hydrodynamics. Here I show that the troublesome inertial term in Eckart's heat flux arises only if one insists on defining thermal diffusivity as a spacetime constant. I argue that this is the most natural definition, and that all confusion disappears if one considers instead th…
▽ More
Does heat have inertia? This question is at the core of a long-standing controversy on Eckart's dissipative relativistic hydrodynamics. Here I show that the troublesome inertial term in Eckart's heat flux arises only if one insists on defining thermal diffusivity as a spacetime constant. I argue that this is the most natural definition, and that all confusion disappears if one considers instead the space-dependent comoving diffusivity, in line with the fact that, in the presence of gravity, space is an inhomogeneous medium.
△ Less
Submitted 15 February, 2012;
originally announced February 2012.
-
Metadiffusion
Authors:
Matteo Smerlak
Abstract:
We predict that the speed of a diffusion front is modified by tidal forces. This effect is small in astrophysical situations, but could be significant - indeed tunably so - in "gravitational analogues" such as gradient-index lenses. Confirming this prediction would extend the concept of metamaterials to the realm of dissipative transport phenomena.
We predict that the speed of a diffusion front is modified by tidal forces. This effect is small in astrophysical situations, but could be significant - indeed tunably so - in "gravitational analogues" such as gradient-index lenses. Confirming this prediction would extend the concept of metamaterials to the realm of dissipative transport phenomena.
△ Less
Submitted 22 January, 2012;
originally announced January 2012.
-
Tailoring diffusion in analogue spacetimes
Authors:
Matteo Smerlak
Abstract:
Diffusive transport is characterized by the scaling law $(length)^{2}\propto(time)$. In this letter we show that this relationship is significantly altered in curved analogue spacetimes. This circumstance provides an opportunity to tailor diffusion: by a suitable design of the analogue metric, it is possible to create materials where diffusion is either faster or slower than in normal media, as de…
▽ More
Diffusive transport is characterized by the scaling law $(length)^{2}\propto(time)$. In this letter we show that this relationship is significantly altered in curved analogue spacetimes. This circumstance provides an opportunity to tailor diffusion: by a suitable design of the analogue metric, it is possible to create materials where diffusion is either faster or slower than in normal media, as desired. This prediction can in principle be tested experimentally with optical analogues, curved graphene sheets, etc. - indeed with any analogue spacetime.
△ Less
Submitted 9 April, 2012; v1 submitted 4 December, 2011;
originally announced December 2011.
-
Diffusion in Curved Spacetimes
Authors:
Matteo Smerlak
Abstract:
Using simple kinematical arguments, we derive the Fokker-Planck equation for diffusion processes in curved spacetimes. In the case of Brownian motion, it coincides with Eckart's relativistic heat equation (albeit in a simpler form), and therefore provides a microscopic justification for his phenomenological heat-flux ansatz. Furthermore, we obtain the small-time asymptotic expansion of the mean sq…
▽ More
Using simple kinematical arguments, we derive the Fokker-Planck equation for diffusion processes in curved spacetimes. In the case of Brownian motion, it coincides with Eckart's relativistic heat equation (albeit in a simpler form), and therefore provides a microscopic justification for his phenomenological heat-flux ansatz. Furthermore, we obtain the small-time asymptotic expansion of the mean square displacement of Brownian motion in static spacetimes. Beyond general relativity itself, this result has potential applications in analogue gravitational systems.
△ Less
Submitted 18 November, 2011; v1 submitted 17 April, 2011;
originally announced April 2011.
