-
An ensemble Kalman approach to randomized maximum likelihood estimation
Authors:
Pavlos Stavrinides,
Elizabeth Qian
Abstract:
This work proposes ensemble Kalman randomized maximum likelihood estimation, a new derivative-free method for performing randomized maximum likelihood estimation, which is a method that can be used to generate approximate samples from posterior distributions in Bayesian inverse problems. The new method has connections to ensemble Kalman inversion and works by evolving an ensemble so that each ense…
▽ More
This work proposes ensemble Kalman randomized maximum likelihood estimation, a new derivative-free method for performing randomized maximum likelihood estimation, which is a method that can be used to generate approximate samples from posterior distributions in Bayesian inverse problems. The new method has connections to ensemble Kalman inversion and works by evolving an ensemble so that each ensemble member solves an instance of a randomly perturbed optimization problem. Linear analysis demonstrates that ensemble members converge exponentially fast to randomized maximum likelihood estimators and, furthermore, that the new method produces samples from the Bayesian posterior when applied to a suitably regularized optimization problem. The method requires that the forward operator, relating the unknown parameter to the data, be evaluated once per iteration per ensemble member, which can be prohibitively expensive when the forward model requires the evolution of a high-dimensional dynamical system. We propose a strategy for making the proposed method tractable in this setting based on a balanced truncation model reduction method tailored to the Bayesian smoothing problem. Theoretical results show near-optimality of this model reduction approach via convergence to an optimal approximation of the posterior covariance as a low-rank update to the prior covariance. Numerical experiments verify theoretical results and illustrate computational acceleration through model reduction.
△ Less
Submitted 3 July, 2025;
originally announced July 2025.
-
Dimension and model reduction approaches for linear Bayesian inverse problems with rank-deficient prior covariances
Authors:
Josie König,
Elizabeth Qian,
Melina A. Freitag
Abstract:
Bayesian inverse problems use observed data to update a prior probability distribution for an unknown state or parameter of a scientific system to a posterior distribution conditioned on the data. In many applications, the unknown parameter is high-dimensional, making computation of the posterior expensive due to the need to sample in a high-dimensional space and the need to evaluate an expensive…
▽ More
Bayesian inverse problems use observed data to update a prior probability distribution for an unknown state or parameter of a scientific system to a posterior distribution conditioned on the data. In many applications, the unknown parameter is high-dimensional, making computation of the posterior expensive due to the need to sample in a high-dimensional space and the need to evaluate an expensive high-dimensional forward model relating the unknown parameter to the data. However, inverse problems often exhibit low-dimensional structure due to the fact that the available data are only informative in a low-dimensional subspace of the parameter space. Dimension reduction approaches exploit this structure by restricting inference to the low-dimensional subspace informed by the data, which can be sampled more efficiently. Further computational cost reductions can be achieved by replacing expensive high-dimensional forward models with cheaper lower-dimensional reduced models. In this work, we propose new dimension and model reduction approaches for linear Bayesian inverse problems with rank-deficient prior covariances, which arise in many practical inference settings. The dimension reduction approach is applicable to general linear Bayesian inverse problems whereas the model reduction approaches are specific to the problem of inferring the initial condition of a linear dynamical system. We provide theoretical approximation guarantees as well as numerical experiments demonstrating the accuracy and efficiency of the proposed approaches.
△ Less
Submitted 30 June, 2025;
originally announced June 2025.
-
The Fundamental Subspaces of Ensemble Kalman Inversion
Authors:
Elizabeth Qian,
Christopher Beattie
Abstract:
Ensemble Kalman Inversion (EKI) methods are a family of iterative methods for solving weighted least-squares problems, especially those arising in scientific and engineering inverse problems in which unknown parameters or states are estimated from observed data by minimizing the weighted square norm of the data misfit. Implementation of EKI requires only evaluation of the forward model mapping the…
▽ More
Ensemble Kalman Inversion (EKI) methods are a family of iterative methods for solving weighted least-squares problems, especially those arising in scientific and engineering inverse problems in which unknown parameters or states are estimated from observed data by minimizing the weighted square norm of the data misfit. Implementation of EKI requires only evaluation of the forward model mapping the unknown to the data, and does not require derivatives or adjoints of the forward model. The methods therefore offer an attractive alternative to gradient-based optimization approaches in inverse problem settings where evaluating derivatives or adjoints of the forward model is computationally intractable. This work presents a new analysis of the behavior of both deterministic and stochastic versions of basic EKI for linear observation operators, resulting in a natural interpretation of EKI's convergence properties in terms of ``fundamental subspaces'' analogous to Strang's fundamental subspaces of linear algebra. Our analysis directly examines the discrete EKI iterations instead of their continuous-time limits considered in previous analyses, and provides spectral decompositions that define six fundamental subspaces of EKI spanning both observation and state spaces. This approach verifies convergence rates previously derived for continuous-time limits, and yields new results describing both deterministic and stochastic EKI convergence behavior with respect to the standard minimum-norm weighted least squares solution in terms of the fundamental subspaces. Numerical experiments illustrate our theoretical results.
△ Less
Submitted 23 May, 2025; v1 submitted 13 September, 2024;
originally announced September 2024.
-
Energy-Preserving Reduced Operator Inference for Efficient Design and Control
Authors:
Tomoki Koike,
Elizabeth Qian
Abstract:
Many-query computations, in which a computational model for an engineering system must be evaluated many times, are crucial in design and control. For systems governed by partial differential equations (PDEs), typical high-fidelity numerical models are high-dimensional and too computationally expensive for the many-query setting. Thus, efficient surrogate models are required to enable low-cost com…
▽ More
Many-query computations, in which a computational model for an engineering system must be evaluated many times, are crucial in design and control. For systems governed by partial differential equations (PDEs), typical high-fidelity numerical models are high-dimensional and too computationally expensive for the many-query setting. Thus, efficient surrogate models are required to enable low-cost computations in design and control. This work presents a physics-preserving reduced model learning approach that targets PDEs whose quadratic operators preserve energy, such as those arising in governing equations in many fluids problems. The approach is based on the Operator Inference method, which fits reduced model operators to state snapshot and time derivative data in a least-squares sense. However, Operator Inference does not generally learn a reduced quadratic operator with the energy-preserving property of the original PDE. Thus, we propose a new energy-preserving Operator Inference (EP-OpInf) approach, which imposes this structure on the learned reduced model via constrained optimization. Numerical results using the viscous Burgers' and Kuramoto-Sivashinksy equation (KSE) demonstrate that EP-OpInf learns efficient and accurate reduced models that retain this energy-preserving structure.
△ Less
Submitted 7 February, 2024; v1 submitted 5 January, 2024;
originally announced January 2024.
-
The Cost-Accuracy Trade-Off In Operator Learning With Neural Networks
Authors:
Maarten V. de Hoop,
Daniel Zhengyu Huang,
Elizabeth Qian,
Andrew M. Stuart
Abstract:
The term `surrogate modeling' in computational science and engineering refers to the development of computationally efficient approximations for expensive simulations, such as those arising from numerical solution of partial differential equations (PDEs). Surrogate modeling is an enabling methodology for many-query computations in science and engineering, which include iterative methods in optimiz…
▽ More
The term `surrogate modeling' in computational science and engineering refers to the development of computationally efficient approximations for expensive simulations, such as those arising from numerical solution of partial differential equations (PDEs). Surrogate modeling is an enabling methodology for many-query computations in science and engineering, which include iterative methods in optimization and sampling methods in uncertainty quantification. Over the last few years, several approaches to surrogate modeling for PDEs using neural networks have emerged, motivated by successes in using neural networks to approximate nonlinear maps in other areas. In principle, the relative merits of these different approaches can be evaluated by understanding, for each one, the cost required to achieve a given level of accuracy. However, the absence of a complete theory of approximation error for these approaches makes it difficult to assess this cost-accuracy trade-off. The purpose of the paper is to provide a careful numerical study of this issue, comparing a variety of different neural network architectures for operator approximation across a range of problems arising from PDE models in continuum mechanics.
△ Less
Submitted 11 August, 2022; v1 submitted 24 March, 2022;
originally announced March 2022.
-
Model Reduction of Linear Dynamical Systems via Balancing for Bayesian Inference
Authors:
Elizabeth Qian,
Jemima M. Tabeart,
Christopher Beattie,
Serkan Gugercin,
Jiahua Jiang,
Peter R. Kramer,
Akil Narayan
Abstract:
We consider the Bayesian approach to the linear Gaussian inference problem of inferring the initial condition of a linear dynamical system from noisy output measurements taken after the initial time. In practical applications, the large dimension of the dynamical system state poses a computational obstacle to computing the exact posterior distribution. Model reduction offers a variety of computati…
▽ More
We consider the Bayesian approach to the linear Gaussian inference problem of inferring the initial condition of a linear dynamical system from noisy output measurements taken after the initial time. In practical applications, the large dimension of the dynamical system state poses a computational obstacle to computing the exact posterior distribution. Model reduction offers a variety of computational tools that seek to reduce this computational burden. In particular, balanced truncation is a system-theoretic approach to model reduction which obtains an efficient reduced-dimension dynamical system by projecting the system operators onto state directions which trade off the reachability and observability of state directions as expressed through the associated Gramians. We introduce Gramian definitions relevant to the inference setting and propose a balanced truncation approach based on these inference Gramians that yield a reduced dynamical system that can be used to cheaply approximate the posterior mean and covariance. Our definitions exploit natural connections between (i) the reachability Gramian and the prior covariance and (ii) the observability Gramian and the Fisher information. The resulting reduced model then inherits stability properties and error bounds from system theoretic considerations, and in some settings yields an optimal posterior covariance approximation. Numerical demonstrations on two benchmark problems in model reduction show that our method can yield near-optimal posterior covariance approximations with order-of-magnitude state dimension reduction.
△ Less
Submitted 25 November, 2021;
originally announced November 2021.
-
Reduced operator inference for nonlinear partial differential equations
Authors:
Elizabeth Qian,
Ionut-Gabriel Farcas,
Karen Willcox
Abstract:
We present a new scientific machine learning method that learns from data a computationally inexpensive surrogate model for predicting the evolution of a system governed by a time-dependent nonlinear partial differential equation (PDE), an enabling technology for many computational algorithms used in engineering settings. Our formulation generalizes to the function space PDE setting the Operator I…
▽ More
We present a new scientific machine learning method that learns from data a computationally inexpensive surrogate model for predicting the evolution of a system governed by a time-dependent nonlinear partial differential equation (PDE), an enabling technology for many computational algorithms used in engineering settings. Our formulation generalizes to the function space PDE setting the Operator Inference method previously developed in [B. Peherstorfer and K. Willcox, Data-driven operator inference for non-intrusive projection-based model reduction, Computer Methods in Applied Mechanics and Engineering, 306 (2016)] for systems governed by ordinary differential equations. The method brings together two main elements. First, ideas from projection-based model reduction are used to explicitly parametrize the learned model by low-dimensional polynomial operators which reflect the known form of the governing PDE. Second, supervised machine learning tools are used to infer from data the reduced operators of this physics-informed parametrization. For systems whose governing PDEs contain more general (non-polynomial) nonlinearities, the learned model performance can be improved through the use of lifting variable transformations, which expose polynomial structure in the PDE. The proposed method is demonstrated on two examples: a heat equation model problem that demonstrates the benefits of the function space formulation in terms of consistency with the underlying continuous truth, and a three-dimensional combustion simulation with over 18 million degrees of freedom, for which the learned reduced models achieve accurate predictions with a dimension reduction of five orders of magnitude and model runtime reduction of up to nine orders of magnitude.
△ Less
Submitted 25 February, 2022; v1 submitted 29 January, 2021;
originally announced February 2021.
-
Lift & Learn: Physics-informed machine learning for large-scale nonlinear dynamical systems
Authors:
Elizabeth Qian,
Boris Kramer,
Benjamin Peherstorfer,
Karen Willcox
Abstract:
We present Lift & Learn, a physics-informed method for learning low-dimensional models for large-scale dynamical systems. The method exploits knowledge of a system's governing equations to identify a coordinate transformation in which the system dynamics have quadratic structure. This transformation is called a lifting map because it often adds auxiliary variables to the system state. The lifting…
▽ More
We present Lift & Learn, a physics-informed method for learning low-dimensional models for large-scale dynamical systems. The method exploits knowledge of a system's governing equations to identify a coordinate transformation in which the system dynamics have quadratic structure. This transformation is called a lifting map because it often adds auxiliary variables to the system state. The lifting map is applied to data obtained by evaluating a model for the original nonlinear system. This lifted data is projected onto its leading principal components, and low-dimensional linear and quadratic matrix operators are fit to the lifted reduced data using a least-squares operator inference procedure. Analysis of our method shows that the Lift & Learn models are able to capture the system physics in the lifted coordinates at least as accurately as traditional intrusive model reduction approaches. This preservation of system physics makes the Lift & Learn models robust to changes in inputs. Numerical experiments on the FitzHugh-Nagumo neuron activation model and the compressible Euler equations demonstrate the generalizability of our model.
△ Less
Submitted 26 March, 2020; v1 submitted 17 December, 2019;
originally announced December 2019.
-
Towards Scalable Parallel-in-Time Turbulent Flow Simulations
Authors:
Qiqi Wang,
Steven Gomez,
Patrick Blonigan,
Alastair Gregory,
Elizabeth Qian
Abstract:
We present a reformulation of unsteady turbulent flow simulations. The initial condition is relaxed and information is allowed to propagate both forward and backward in time. Simulations of chaotic dynamical systems with this reformulation can be proven to be well-conditioned time domain boundary value problems. The reformulation can enable scalable parallel-in-time simulation of turbulent flows.
We present a reformulation of unsteady turbulent flow simulations. The initial condition is relaxed and information is allowed to propagate both forward and backward in time. Simulations of chaotic dynamical systems with this reformulation can be proven to be well-conditioned time domain boundary value problems. The reformulation can enable scalable parallel-in-time simulation of turbulent flows.
△ Less
Submitted 18 July, 2013; v1 submitted 11 November, 2012;
originally announced November 2012.
