-
When are dynamical systems learned from time series data statistically accurate?
Authors:
Jeongjin Park,
Nicole Yang,
Nisha Chandramoorthy
Abstract:
Conventional notions of generalization often fail to describe the ability of learned models to capture meaningful information from dynamical data. A neural network that learns complex dynamics with a small test error may still fail to reproduce its \emph{physical} behavior, including associated statistical moments and Lyapunov exponents. To address this gap, we propose an ergodic theoretic approac…
▽ More
Conventional notions of generalization often fail to describe the ability of learned models to capture meaningful information from dynamical data. A neural network that learns complex dynamics with a small test error may still fail to reproduce its \emph{physical} behavior, including associated statistical moments and Lyapunov exponents. To address this gap, we propose an ergodic theoretic approach to generalization of complex dynamical models learned from time series data. Our main contribution is to define and analyze generalization of a broad suite of neural representations of classes of ergodic systems, including chaotic systems, in a way that captures emulating underlying invariant, physical measures. Our results provide theoretical justification for why regression methods for generators of dynamical systems (Neural ODEs) fail to generalize, and why their statistical accuracy improves upon adding Jacobian information during training. We verify our results on a number of ergodic chaotic systems and neural network parameterizations, including MLPs, ResNets, Fourier Neural layers, and RNNs.
△ Less
Submitted 17 June, 2025; v1 submitted 9 November, 2024;
originally announced November 2024.
-
Score Operator Newton transport
Authors:
Nisha Chandramoorthy,
Florian Schaefer,
Youssef Marzouk
Abstract:
We propose a new approach for sampling and Bayesian computation that uses the score of the target distribution to construct a transport from a given reference distribution to the target. Our approach is an infinite-dimensional Newton method, involving a linear PDE, for finding a zero of a ``score-residual'' operator. We prove sufficient conditions for convergence to a valid transport map. Our Newt…
▽ More
We propose a new approach for sampling and Bayesian computation that uses the score of the target distribution to construct a transport from a given reference distribution to the target. Our approach is an infinite-dimensional Newton method, involving a linear PDE, for finding a zero of a ``score-residual'' operator. We prove sufficient conditions for convergence to a valid transport map. Our Newton iterates can be computed by exploiting fast solvers for elliptic PDEs, resulting in new algorithms for Bayesian inference and other sampling tasks. We identify elementary settings where score-operator Newton transport achieves fast convergence while avoiding mode collapse.
△ Less
Submitted 9 March, 2024; v1 submitted 16 May, 2023;
originally announced May 2023.
-
On the generalization of learning algorithms that do not converge
Authors:
Nisha Chandramoorthy,
Andreas Loukas,
Khashayar Gatmiry,
Stefanie Jegelka
Abstract:
Generalization analyses of deep learning typically assume that the training converges to a fixed point. But, recent results indicate that in practice, the weights of deep neural networks optimized with stochastic gradient descent often oscillate indefinitely. To reduce this discrepancy between theory and practice, this paper focuses on the generalization of neural networks whose training dynamics…
▽ More
Generalization analyses of deep learning typically assume that the training converges to a fixed point. But, recent results indicate that in practice, the weights of deep neural networks optimized with stochastic gradient descent often oscillate indefinitely. To reduce this discrepancy between theory and practice, this paper focuses on the generalization of neural networks whose training dynamics do not necessarily converge to fixed points. Our main contribution is to propose a notion of statistical algorithmic stability (SAS) that extends classical algorithmic stability to non-convergent algorithms and to study its connection to generalization. This ergodic-theoretic approach leads to new insights when compared to the traditional optimization and learning theory perspectives. We prove that the stability of the time-asymptotic behavior of a learning algorithm relates to its generalization and empirically demonstrate how loss dynamics can provide clues to generalization performance. Our findings provide evidence that networks that "train stably generalize better" even when the training continues indefinitely and the weights do not converge.
△ Less
Submitted 19 August, 2022; v1 submitted 16 August, 2022;
originally announced August 2022.
-
Rigorous justification for the space-split sensitivity algorithm to compute linear response in Anosov systems
Authors:
Nisha Chandramoorthy,
Malo Jézéquel
Abstract:
Ruelle gave a formula for linear response of transitive Anosov diffeomorphisms. Recently, practically computable realizations of Ruelle's formula have emerged that potentially enable sensitivity analysis of certain high-dimensional chaotic numerical simulations encountered in the applied sciences. In this paper, we provide full mathematical justification for the convergence of one such efficient c…
▽ More
Ruelle gave a formula for linear response of transitive Anosov diffeomorphisms. Recently, practically computable realizations of Ruelle's formula have emerged that potentially enable sensitivity analysis of certain high-dimensional chaotic numerical simulations encountered in the applied sciences. In this paper, we provide full mathematical justification for the convergence of one such efficient computation, the space-split sensitivity, or S3, algorithm. In S3, Ruelle's formula is computed as a sum of two terms obtained by decomposing the perturbation vector field into a coboundary and a remainder that is parallel to the unstable direction. Such a decomposition results in a splitting of Ruelle's formula that is amenable to efficient computation. We prove the existence of the S3 decomposition and the convergence of the computations of both resulting components of Ruelle's formula.
△ Less
Submitted 5 July, 2023; v1 submitted 6 September, 2021;
originally announced September 2021.
-
Efficient computation of linear response of chaotic attractors with one-dimensional unstable manifolds
Authors:
Nisha Chandramoorthy,
Qiqi Wang
Abstract:
This paper presents the space-split sensitivity or the S3 algorithm to transform Ruelle's linear response formula into a well-conditioned ergodic-averaging computation. We prove a decomposition of Ruelle's formula that is differentiable on the unstable manifold, which we assume to be one-dimensional. This decomposition of Ruelle's formula ensures that one of the resulting terms, the stable contrib…
▽ More
This paper presents the space-split sensitivity or the S3 algorithm to transform Ruelle's linear response formula into a well-conditioned ergodic-averaging computation. We prove a decomposition of Ruelle's formula that is differentiable on the unstable manifold, which we assume to be one-dimensional. This decomposition of Ruelle's formula ensures that one of the resulting terms, the stable contribution, can be computed using a regularized tangent equation, similar to in a non-chaotic system. The remaining term, known as the unstable contribution, is regularized and converted into an efficiently computable ergodic average. In this process, we develop new algorithms, which may be useful beyond linear response, to compute i) a fundamental statistical quantity we introduce called the density gradient, and ii) the unstable derivatives of the regularized tangent vector field and the unstable direction. We prove that the S3 algorithm, which combines these computational ingredients that enter the stable and unstable contribution, converges like a Monte Carlo approximation of Ruelle's formula. The algorithm presented here is hence a first step toward full-fledged applications of sensitivity analysis in chaotic systems, wherever such applications have been limited due to lack of availability of long-term sensitivities.
△ Less
Submitted 5 January, 2022; v1 submitted 15 March, 2021;
originally announced March 2021.
-
Computational assessment of smooth and rough parameter dependence of statistics in chaotic dynamical systems
Authors:
Adam A. Sliwiak,
Nisha Chandramoorthy,
Qiqi Wang
Abstract:
An assumption of smooth response to small parameter changes, of statistics or long-time averages of a chaotic system, is generally made in the field of sensitivity analysis, and the parametric derivatives of statistical quantities are critically used in science and engineering. In this paper, we propose a numerical procedure to assess the differentiability of statistics with respect to parameters…
▽ More
An assumption of smooth response to small parameter changes, of statistics or long-time averages of a chaotic system, is generally made in the field of sensitivity analysis, and the parametric derivatives of statistical quantities are critically used in science and engineering. In this paper, we propose a numerical procedure to assess the differentiability of statistics with respect to parameters in chaotic systems. We numerically show that the existence of the derivative depends on the Lebesgue-integrability of a certain density gradient function, which we define as the derivative of logarithmic SRB density along the unstable manifold. We develop a recursive formula for the density gradient that can be efficiently computed along trajectories, and demonstrate its use in determining the differentiability of statistics. Our numerical procedure is illustrated on low-dimensional chaotic systems whose statistics exhibit both smooth and rough regions in parameter space.
△ Less
Submitted 25 January, 2021; v1 submitted 20 January, 2021;
originally announced January 2021.
-
Variational optimization and data assimilation in chaotic time-delayed systems with automatic-differentiated shadowing sensitivity
Authors:
Nisha Chandramoorthy,
Luca Magri,
Qiqi Wang
Abstract:
In this computational paper, we perform sensitivity analysis of long-time (or ensemble) averages in the chaotic regime using the shadowing algorithm. We introduce automatic differentiation to eliminate the tangent/adjoint equation solvers used in the shadowing algorithm. In a gradient-based optimization, we use the computed shadowing sensitivity to minimize different long-time averaged functionals…
▽ More
In this computational paper, we perform sensitivity analysis of long-time (or ensemble) averages in the chaotic regime using the shadowing algorithm. We introduce automatic differentiation to eliminate the tangent/adjoint equation solvers used in the shadowing algorithm. In a gradient-based optimization, we use the computed shadowing sensitivity to minimize different long-time averaged functionals of a chaotic time-delayed system by optimal parameter selection. In combined state and parameter estimation for data assimilation, we use the computed sensitivity to predict the optimal trajectory given information from a model and data from measurements beyond the predictability time. The algorithms are applied to a thermoacoustic model. Because the computational framework is rather general, the techniques presented in this paper may be used for sensitivity analysis of ensemble averages, parameter optimization and data assimilation of other chaotic problems, where shadowing methods are applicable.
△ Less
Submitted 16 November, 2020;
originally announced November 2020.
-
On the probability of finding a nonphysical solution through shadowing
Authors:
Nisha Chandramoorthy,
Qiqi Wang
Abstract:
This paper proves that shadowing solutions can be almost surely nonphysical. This finding invalidates the argument that small perturbations in a chaotic system can only have a small impact on its statistical behavior. This theoretical finding has implications for many applications in which chaotic mechanics plays an important role. It suggests, for example, that we can control the climate through…
▽ More
This paper proves that shadowing solutions can be almost surely nonphysical. This finding invalidates the argument that small perturbations in a chaotic system can only have a small impact on its statistical behavior. This theoretical finding has implications for many applications in which chaotic mechanics plays an important role. It suggests, for example, that we can control the climate through subtle perturbations. It also suggests that numerical simulations of chaotic dynamics, such as turbulent flows, may fail to predict the true long-term or statistical behavior.
△ Less
Submitted 21 November, 2020; v1 submitted 26 October, 2020;
originally announced October 2020.
-
An ergodic averaging method to differentiate covariant Lyapunov vectors
Authors:
Nisha Chandramoorthy,
Qiqi Wang
Abstract:
Covariant Lyapunov vectors or CLVs span the expanding and contracting directions of perturbations along trajectories in a chaotic dynamical system. Due to efficient algorithms to compute them that only utilize trajectory information, they have been widely applied across scientific disciplines, principally for sensitivity analysis and predictions under uncertainty. In this paper, we develop a numer…
▽ More
Covariant Lyapunov vectors or CLVs span the expanding and contracting directions of perturbations along trajectories in a chaotic dynamical system. Due to efficient algorithms to compute them that only utilize trajectory information, they have been widely applied across scientific disciplines, principally for sensitivity analysis and predictions under uncertainty. In this paper, we develop a numerical method to compute the directional derivatives of CLVs along their own directions. Similar to the computation of CLVs, the present method for their derivatives is iterative and analogously uses the second-order derivative of the chaotic map along trajectories, in addition to the Jacobian. We validate the new method on a super-contracting Smale-Williams Solenoid attractor. We also demonstrate the algorithm on several other examples including smoothly perturbed Arnold Cat maps, and the Lorenz attractor, obtaining visualizations of the curvature of each attractor. Furthermore, we reveal a fundamental connection of the CLV self-derivatives with a statistical linear response formula.
△ Less
Submitted 26 April, 2021; v1 submitted 16 July, 2020;
originally announced July 2020.
-
A computable realization of Ruelle's formula for linear response of statistics in chaotic systems
Authors:
Nisha Chandramoorthy,
Qiqi Wang
Abstract:
We present a computable reformulation of Ruelle's linear response formula for chaotic systems. The new formula, called Space-Split Sensitivity or S3, achieves an error convergence of the order ${\cal O}(1/\sqrt{N})$ using $N$ phase points. The reformulation is based on splitting the overall sensitivity into that to stable and unstable components of the perturbation. The unstable contribution to th…
▽ More
We present a computable reformulation of Ruelle's linear response formula for chaotic systems. The new formula, called Space-Split Sensitivity or S3, achieves an error convergence of the order ${\cal O}(1/\sqrt{N})$ using $N$ phase points. The reformulation is based on splitting the overall sensitivity into that to stable and unstable components of the perturbation. The unstable contribution to the sensitivity is regularized using ergodic properties and the hyperbolic structure of the dynamics. Numerical examples of uniformly hyperbolic attractors are used to validate the S3 formula against a naïve finite-difference calculation; sensitivities match closely, with far fewer sample points required by S3.
△ Less
Submitted 8 June, 2020; v1 submitted 10 February, 2020;
originally announced February 2020.
-
Feasibility analysis of ensemble sensitivity computation in turbulent flows
Authors:
Nisha Chandramoorthy,
Pablo Fernandez,
Chaitanya Talnikar,
Qiqi Wang
Abstract:
In chaotic systems, such as turbulent flows, the solutions to tangent and adjoint equations exhibit an unbounded growth in their norms. This behavior renders the instantaneous tangent and adjoint solutions unusable for sensitivity analysis. The Lea-Allen-Haine ensemble sensitivity (ES) estimates provide a way of computing meaningful sensitivities in chaotic systems by utilizing tangent/adjoint sol…
▽ More
In chaotic systems, such as turbulent flows, the solutions to tangent and adjoint equations exhibit an unbounded growth in their norms. This behavior renders the instantaneous tangent and adjoint solutions unusable for sensitivity analysis. The Lea-Allen-Haine ensemble sensitivity (ES) estimates provide a way of computing meaningful sensitivities in chaotic systems by utilizing tangent/adjoint solutions over short trajectories. In this paper, we analyze the feasibility of ES computations under optimistic mathematical assumptions on the flow dynamics. Furthermore, we estimate upper bounds on the rate of convergence of the ES method in numerical simulations of turbulent flow. Even at the optimistic upper bound, the ES method is computationally intractable in each of the numerical examples considered.
△ Less
Submitted 13 July, 2019; v1 submitted 19 November, 2018;
originally announced November 2018.
