Showing 1–50 of 61 results for author: Mehta, P

Dual Filter: A Mathematical Framework for Inference using Transformer-like Architectures

Authors: Heng-Sheng Chang, Prashant G. Mehta

Abstract: This paper presents a mathematical framework for causal nonlinear prediction in settings where observations are generated from an underlying hidden Markov model (HMM). Both the problem formulation and the proposed solution are motivated by the decoder-only transformer architecture, in which a finite sequence of observations (tokens) is mapped to the conditional probability of the next token. Our o… ▽ More This paper presents a mathematical framework for causal nonlinear prediction in settings where observations are generated from an underlying hidden Markov model (HMM). Both the problem formulation and the proposed solution are motivated by the decoder-only transformer architecture, in which a finite sequence of observations (tokens) is mapped to the conditional probability of the next token. Our objective is not to construct a mathematical model of a transformer. Rather, our interest lies in deriving, from first principles, transformer-like architectures that solve the prediction problem for which the transformer is designed. The proposed framework is based on an original optimal control approach, where the prediction objective (MMSE) is reformulated as an optimal control problem. An analysis of the optimal control problem is presented leading to a fixed-point equation on the space of probability measures. To solve the fixed-point equation, we introduce the dual filter, an iterative algorithm that closely parallels the architecture of decoder-only transformers. These parallels are discussed in detail along with the relationship to prior work on mathematical modeling of transformers as transport on the space of probability measures. Numerical experiments are provided to illustrate the performance of the algorithm using parameter values used in researchscale transformer models. △ Less

Submitted 1 May, 2025; originally announced May 2025.

Comments: 49 pages, 6 figures

Error Analysis of Sampling Algorithms for Approximating Stochastic Optimal Control

Authors: Anant A. Joshi, Amirhossein Taghvaei, Prashant G. Mehta

Abstract: This paper is concerned with the error analysis of two types of sampling algorithms, namely model predictive path integral (MPPI) and an interacting particle system (\IPS) algorithm, that have been proposed in the literature for numerical approximation of the stochastic optimal control. The analysis is presented through the lens of Gibbs variational principle. For an illustrative example of a sing… ▽ More This paper is concerned with the error analysis of two types of sampling algorithms, namely model predictive path integral (MPPI) and an interacting particle system (\IPS) algorithm, that have been proposed in the literature for numerical approximation of the stochastic optimal control. The analysis is presented through the lens of Gibbs variational principle. For an illustrative example of a single-stage stochastic optimal control problem, analytical expressions for approximation error and scaling laws, with respect to the state dimension and sample size, are derived. The analytical results are illustrated with numerical simulations. △ Less

Submitted 2 April, 2025; originally announced April 2025.

Functional role of synchronization: A mean-field control perspective

Authors: Prashant Mehta, Sean Meyn

Abstract: The broad goal of the research surveyed in this article is to develop methods for understanding the aggregate behavior of interconnected dynamical systems, as found in mathematical physics, neuroscience, economics, power systems and neural networks. Questions concern prediction of emergent (often unanticipated) phenomena, methods to formulate distributed control schemes to influence this behavior,… ▽ More The broad goal of the research surveyed in this article is to develop methods for understanding the aggregate behavior of interconnected dynamical systems, as found in mathematical physics, neuroscience, economics, power systems and neural networks. Questions concern prediction of emergent (often unanticipated) phenomena, methods to formulate distributed control schemes to influence this behavior, and these topics prompt many other questions in the domain of learning. The area of mean field games, pioneered by Peter Caines, are well suited to addressing these topics. The approach is surveyed in the present paper within the context of controlled coupled oscillators. △ Less

Submitted 1 February, 2025; originally announced February 2025.

Comments: To appear, J Syst Sci Complex

MSC Class: 49N80; 68T05

How to implement the Bayes' formula in the age of ML?

Authors: Amirhossein Taghvaei, Prashant G. Mehta

Abstract: This chapter contains a self-contained introduction to the significance of Bayes' formula in the context of nonlinear filtering problems. Both discrete-time and continuous-time settings of the problem are considered in a unified manner. In control theory, the focus on optimization-based solution approaches is stressed together with a discussion of historical developments in this area (from 1960s o… ▽ More This chapter contains a self-contained introduction to the significance of Bayes' formula in the context of nonlinear filtering problems. Both discrete-time and continuous-time settings of the problem are considered in a unified manner. In control theory, the focus on optimization-based solution approaches is stressed together with a discussion of historical developments in this area (from 1960s onwards). The heart of this chapter contains a presentation of a novel optimal transportation formulation for the Bayes formula (developed recently by the first author) and its relationship to some of the prior joint work (feedback particle filter) from the authors. The presentation highlights how optimal transportation theory is leveraged to overcome some of the numerical challenges of implementing Bayes' law by enabling the use of machine learning (ML) tools. △ Less

Submitted 14 November, 2024; originally announced November 2024.

Jacobi convolution series for Petrov-Galerkin scheme and general fractional calculus of arbitrary order over finite interval

Authors: Pavan Pranjivan Mehta, Gianluigi Rozza

Abstract: Recently, general fractional calculus was introduced by Kochubei (2011) and Luchko (2021) as a further generalisation of fractional calculus, where the derivative and integral operator admits arbitrary kernel. Such a formalism will have many applications in physics and engineering, since the kernel is no longer restricted. We first extend the work of Al-Refai and Luchko (2023) on finite interval t… ▽ More Recently, general fractional calculus was introduced by Kochubei (2011) and Luchko (2021) as a further generalisation of fractional calculus, where the derivative and integral operator admits arbitrary kernel. Such a formalism will have many applications in physics and engineering, since the kernel is no longer restricted. We first extend the work of Al-Refai and Luchko (2023) on finite interval to arbitrary orders. Followed by, developing an efficient Petrov-Galerkin scheme by introducing Jacobi convolution series as basis functions. A notable property of this basis function, the general fractional derivative of Jacobi convolution series is a shifted Jacobi polynomial. Thus, with a suitable test function it results in diagonal stiffness matrix, hence, the efficiency in implementation. Furthermore, our method is constructed for any arbitrary kernel including that of fractional operator, since, its a special case of general fractional operator. △ Less

Submitted 27 January, 2025; v1 submitted 12 November, 2024; originally announced November 2024.

MSC Class: (2020) 26A33; 35R11; 65M70; 65N35; 33C45; 42C05

Arrow of Time in Estimation and Control: Duality Theory Beyond the Linear Gaussian Model

Authors: Jin Won Kim, Prashant G. Mehta

Abstract: Duality between estimation and control is a foundational concept in Control Theory. Most students learn about the elementary duality -- between observability and controllability -- in their first graduate course in linear systems theory. Therefore, it comes as a surprise that for a more general class of nonlinear stochastic systems (hidden Markov models or HMMs), duality is incomplete. Our objec… ▽ More Duality between estimation and control is a foundational concept in Control Theory. Most students learn about the elementary duality -- between observability and controllability -- in their first graduate course in linear systems theory. Therefore, it comes as a surprise that for a more general class of nonlinear stochastic systems (hidden Markov models or HMMs), duality is incomplete. Our objective in writing this article is two-fold: (i) To describe the difficulty in extending duality to HMMs; and (ii) To discuss its recent resolution by the authors. A key message is that the main difficulty in extending duality comes from time reversal in going from estimation to control. The reason for time reversal is explained with the aid of the familiar linear deterministic and linear Gaussian models. The explanation is used to motivate the difference between the linear and the nonlinear models. Once the difference is understood, duality for HMMs is described based on our recent work. The article also includes a comparison and discussion of the different types of duality considered in literature. △ Less

Submitted 8 October, 2024; v1 submitted 13 May, 2024; originally announced May 2024.

Backward Map for Filter Stability Analysis

Authors: Jin Won Kim, Anant A. Joshi, Prashant G. Mehta

Abstract: In this paper, a backward map is introduced for the purposes of analysis of the nonlinear (stochastic) filter stability. The backward map is important because the filter-stability in the sense of $\chisq$-divergence follows from showing a certain variance decay property for the backward map. To show this property requires additional assumptions on the model properties of the hidden Markov model (H… ▽ More In this paper, a backward map is introduced for the purposes of analysis of the nonlinear (stochastic) filter stability. The backward map is important because the filter-stability in the sense of $\chisq$-divergence follows from showing a certain variance decay property for the backward map. To show this property requires additional assumptions on the model properties of the hidden Markov model (HMM). The analysis in this paper is based on introducing a Poincaré Inequality (PI) for HMMs with white noise observations. In finite state-space settings, PI is related to both the ergodicity of the Markov process as well as the observability of the HMM. It is shown that the Poincaré constant is positive if and only if the HMM is detectable. △ Less

Submitted 8 October, 2024; v1 submitted 2 May, 2024; originally announced May 2024.

Comments: Conference proceeding related to arXiv:2305.12850

Divergence metrics in the study of Markov and hidden Markov processes

Authors: Jin Won Kim, Amirhossein Taghvaei, Prashant G. Mehta

Abstract: This paper is divided into two parts. The first part reviews the formulae for f-divergences in the study of continuous-time Markov processes and explores their applications in areas such as stochastic stability, the second law of thermodynamics, and its non-equilibrium extensions. This sets the foundation for the second part, which focuses on f-divergence in the study of hidden Markov processes. I… ▽ More This paper is divided into two parts. The first part reviews the formulae for f-divergences in the study of continuous-time Markov processes and explores their applications in areas such as stochastic stability, the second law of thermodynamics, and its non-equilibrium extensions. This sets the foundation for the second part, which focuses on f-divergence in the study of hidden Markov processes. In this context, we present analyses of filter stability and stochastic thermodynamics, with the latter being used to illustrate the concept of a Maxwell demon in an over-damped Langevin model with white noise observations. The paper's expository style and unified formalism for both Markov and hidden Markov processes aim to serve as a valuable resource for researchers working across related fields. △ Less

Submitted 2 October, 2024; v1 submitted 24 April, 2024; originally announced April 2024.

Variance Decay Property for Filter Stability

Authors: Jin Won Kim, Prashant G. Mehta

Abstract: This paper is concerned with the problem of nonlinear (stochastic) filter stability of a hidden Markov model (HMM) with white noise observations. A contribution is the variance decay property which is used to conclude filter stability. For this purpose, a new notion of the Poincaré inequality (PI) is introduced for the nonlinear filter. PI is related to both the ergodicity of the Markov process as… ▽ More This paper is concerned with the problem of nonlinear (stochastic) filter stability of a hidden Markov model (HMM) with white noise observations. A contribution is the variance decay property which is used to conclude filter stability. For this purpose, a new notion of the Poincaré inequality (PI) is introduced for the nonlinear filter. PI is related to both the ergodicity of the Markov process as well as the observability of the HMM. The proofs are based upon a recently discovered minimum variance duality which is used to transform the nonlinear filtering problem into a stochastic optimal control problem for a backward stochastic differential equation (BSDE). △ Less

Submitted 26 June, 2024; v1 submitted 22 May, 2023; originally announced May 2023.

Comments: 16 pages

Journal ref: IEEE Transactions on Automatic Control, 2024

A Survey of Feedback Particle Filter and related Controlled Interacting Particle Systems (CIPS)

Authors: Amirhossein Taghvaei, Prashant G. Mehta

Abstract: In this survey, we describe controlled interacting particle systems (CIPS) to approximate the solution of the optimal filtering and the optimal control problems. Part I of the survey is focussed on the feedback particle filter (FPF) algorithm, its derivation based on optimal transportation theory, and its relationship to the ensemble Kalman filter (EnKF) and the conventional sequential importance… ▽ More In this survey, we describe controlled interacting particle systems (CIPS) to approximate the solution of the optimal filtering and the optimal control problems. Part I of the survey is focussed on the feedback particle filter (FPF) algorithm, its derivation based on optimal transportation theory, and its relationship to the ensemble Kalman filter (EnKF) and the conventional sequential importance sampling-resampling (SIR) particle filters. The central numerical problem of FPF -- to approximate the solution of the Poisson equation -- is described together with the main solution approaches. An analytical and numerical comparison with the SIR particle filter is given to illustrate the advantages of the CIPS approach. Part II of the survey is focussed on adapting these algorithms for the problem of reinforcement learning. The survey includes several remarks that describe extensions as well as open problems in this subject. △ Less

Submitted 20 March, 2023; v1 submitted 2 January, 2023; originally announced January 2023.

Sufficient Exploration for Convex Q-learning

Authors: Fan Lu, Prashant Mehta, Sean Meyn, Gergely Neu

Abstract: In recent years there has been a collective research effort to find new formulations of reinforcement learning that are simultaneously more efficient and more amenable to analysis. This paper concerns one approach that builds on the linear programming (LP) formulation of optimal control of Manne. A primal version is called logistic Q-learning, and a dual variant is convex Q-learning. This paper fo… ▽ More In recent years there has been a collective research effort to find new formulations of reinforcement learning that are simultaneously more efficient and more amenable to analysis. This paper concerns one approach that builds on the linear programming (LP) formulation of optimal control of Manne. A primal version is called logistic Q-learning, and a dual variant is convex Q-learning. This paper focuses on the latter, while building bridges with the former. The main contributions follow: (i) The dual of convex Q-learning is not precisely Manne's LP or a version of logistic Q-learning, but has similar structure that reveals the need for regularization to avoid over-fitting. (ii) A sufficient condition is obtained for a bounded solution to the Q-learning LP. (iii) Simulation studies reveal numerical challenges when addressing sampled-data systems based on a continuous time model. The challenge is addressed using state-dependent sampling. The theory is illustrated with applications to examples from OpenAI gym. It is shown that convex Q-learning is successful in cases where standard Q-learning diverges, such as the LQR problem. △ Less

Submitted 17 October, 2022; originally announced October 2022.

Duality for Nonlinear Filtering II: Optimal Control

Authors: Jin Won Kim, Prashant G. Mehta

Abstract: This paper is concerned with the development and use of duality theory for a nonlinear filtering model with white noise observations. The main contribution of this paper is to introduce a stochastic optimal control problem as a dual to the nonlinear filtering problem. The mathematical statement of the dual relationship between the two problems is given in the form of a duality principle. The const… ▽ More This paper is concerned with the development and use of duality theory for a nonlinear filtering model with white noise observations. The main contribution of this paper is to introduce a stochastic optimal control problem as a dual to the nonlinear filtering problem. The mathematical statement of the dual relationship between the two problems is given in the form of a duality principle. The constraint for the optimal control problem is the backward stochastic differential equation (BSDE) introduced in the companion paper. The optimal control solution is obtained from an application of the maximum principle, and subsequently used to derive the equation of the nonlinear filter. The proposed duality is shown to be an exact extension of the classical Kalman-Bucy duality, and different from other types of optimal control and variational formulations given in literature. △ Less

Submitted 13 August, 2022; originally announced August 2022.

Duality for Nonlinear Filtering I: Observability

Authors: Jin Won Kim, Prashant G. Mehta

Abstract: This paper is concerned with the development and use of duality theory for a hidden Markov model (HMM) with white noise observations. The main contribution of this work is to introduce a backward stochastic differential equation (BSDE) as a dual control system. A key outcome is that stochastic observability (resp. detectability) of the HMM is expressed in dual terms: as controllability (resp. stab… ▽ More This paper is concerned with the development and use of duality theory for a hidden Markov model (HMM) with white noise observations. The main contribution of this work is to introduce a backward stochastic differential equation (BSDE) as a dual control system. A key outcome is that stochastic observability (resp. detectability) of the HMM is expressed in dual terms: as controllability (resp. stabilizability) of the dual control system. All aspects of controllability, namely, definition of controllable space and controllability gramian, along with their properties and explicit formulae, are discussed. The proposed duality is shown to be an exact extension of the classical duality in linear systems theory. One can then relate and compare the linear and the nonlinear systems. A side-by-side summary of this relationship is given in a tabular form (Table~II). △ Less

Submitted 13 August, 2022; originally announced August 2022.

Comments: arXiv admin note: text overlap with arXiv:2207.07709

How does a Rational Agent Act in an Epidemic?

Authors: S. Yagiz Olmez, Shubham Aggarwal, Jin Won Kim, Erik Miehling, Tamer Başar, Matthew West, Prashant G. Mehta

Abstract: Evolution of disease in a large population is a function of the top-down policy measures from a centralized planner, as well as the self-interested decisions (to be socially active) of individual agents in a large heterogeneous population. This paper is concerned with understanding the latter based on a mean-field type optimal control model. Specifically, the model is used to investigate the role… ▽ More Evolution of disease in a large population is a function of the top-down policy measures from a centralized planner, as well as the self-interested decisions (to be socially active) of individual agents in a large heterogeneous population. This paper is concerned with understanding the latter based on a mean-field type optimal control model. Specifically, the model is used to investigate the role of partial information on an agent's decision-making, and study the impact of such decisions by a large number of agents on the spread of the virus in the population. The motivation comes from the presymptomatic and asymptomatic spread of the COVID-19 virus where an agent unwittingly spreads the virus. We show that even in a setting with fully rational agents, limited information on the viral state can result in an epidemic growth. △ Less

Submitted 5 June, 2022; originally announced June 2022.

Comments: arXiv admin note: text overlap with arXiv:2111.10422

Modeling Presymptomatic Spread in Epidemics via Mean-Field Games

Authors: S. Yagiz Olmez, Shubham Aggarwal, Jin Won Kim, Erik Miehling, Tamer Başar, Matthew West, Prashant G. Mehta

Abstract: This paper is concerned with developing mean-field game models for the evolution of epidemics. Specifically, an agent's decision -- to be socially active in the midst of an epidemic -- is modeled as a mean-field game with health-related costs and activity-related rewards. By considering the fully and partially observed versions of this problem, the role of information in guiding an agent's rationa… ▽ More This paper is concerned with developing mean-field game models for the evolution of epidemics. Specifically, an agent's decision -- to be socially active in the midst of an epidemic -- is modeled as a mean-field game with health-related costs and activity-related rewards. By considering the fully and partially observed versions of this problem, the role of information in guiding an agent's rational decision is highlighted. The main contributions of the paper are to derive the equations for the mean-field game in both fully and partially observed settings of the problem, to present a complete analysis of the fully observed case, and to present some analytical results for the partially observed case. △ Less

Submitted 19 November, 2021; originally announced November 2021.

A Dynamic Programming Formulation for the Nonlinear Filter

Authors: Jin Won Kim, Prashant G. Mehta

Abstract: This paper build on our recent work where we presented a dual stochastic optimal control formulation of the nonlinear filtering problem [1]. The constraint for the dual problem is a backward stochastic differential equations (BSDE). The solution is obtained via an application of the maximum principle (MP). In the present paper, a dynamic programming (DP) principle is presented for a special class… ▽ More This paper build on our recent work where we presented a dual stochastic optimal control formulation of the nonlinear filtering problem [1]. The constraint for the dual problem is a backward stochastic differential equations (BSDE). The solution is obtained via an application of the maximum principle (MP). In the present paper, a dynamic programming (DP) principle is presented for a special class of BSDE-constrained stochastic optimal control problems. The principle is applied to derive the solution of the nonlinear filtering problem. △ Less

Submitted 29 October, 2021; originally announced November 2021.

Controlled Interacting Particle Algorithms for Simulation-based Reinforcement Learning

Authors: Anant Joshi, Amirhossein Taghvaei, Prashant G. Mehta, Sean P. Meyn

Abstract: This paper is concerned with optimal control problems for control systems in continuous time, and interacting particle system methods designed to construct approximate control solutions. Particular attention is given to the linear quadratic (LQ) control problem. There is a growing interest in re-visiting this classical problem, in part due to the successes of reinforcement learning (RL). The main… ▽ More This paper is concerned with optimal control problems for control systems in continuous time, and interacting particle system methods designed to construct approximate control solutions. Particular attention is given to the linear quadratic (LQ) control problem. There is a growing interest in re-visiting this classical problem, in part due to the successes of reinforcement learning (RL). The main question of this body of research (and also of our paper) is to approximate the optimal control law {\em without} explicitly solving the Riccati equation. A novel simulation-based algorithm, namely a dual ensemble Kalman filter (EnKF), is introduced. The algorithm is used to obtain formulae for optimal control, expressed entirely in terms of the EnKF particles. An extension to the nonlinear case is also presented. The theoretical results and algorithms are illustrated with numerical experiments. △ Less

Submitted 7 July, 2022; v1 submitted 2 July, 2021; originally announced July 2021.

A Dual Characterization of the Stability of the Wonham Filter

Authors: Jin Won Kim, Prashant G. Mehta

Abstract: This paper revisits the classical question of the stability of the nonlinear Wonham filter. The novel contributions of this paper are two-fold: (i) definition of the stabilizability for the (control-theoretic) dual to the nonlinear filter; and (ii) the use of this definition to obtain conclusions on the stability of the Wonham filter. Specifically, it is shown that the stabilizability of the dual… ▽ More This paper revisits the classical question of the stability of the nonlinear Wonham filter. The novel contributions of this paper are two-fold: (i) definition of the stabilizability for the (control-theoretic) dual to the nonlinear filter; and (ii) the use of this definition to obtain conclusions on the stability of the Wonham filter. Specifically, it is shown that the stabilizability of the dual system is necessary for filter stability and conversely stabilizability implies that the filter asymptotically detects the correct ergodic class. The formulation and the proofs are based upon a recently discovered duality result whereby the nonlinear filtering problem is cast as a stochastic optimal control problem for a backward stochastic differential equation (BSDE). The control-theoretic proof techniques and results may be viewed as a generalization of the classical work on the stability of the Kalman filter. △ Less

Submitted 8 October, 2021; v1 submitted 26 March, 2021; originally announced March 2021.

Comments: 60th IEEE Conference on Decision and Control (CDC)

The Conditional Poincaré Inequality for Filter Stability

Authors: Jin Won Kim, Prashant G. Mehta, Sean Meyn

Abstract: This paper is concerned with the problem of nonlinear filter stability of ergodic Markov processes. The main contribution is the conditional Poincaré inequality (PI), which is shown to yield filter stability. The proof is based upon a recently discovered duality which is used to transform the nonlinear filtering problem into a stochastic optimal control problem for a backward stochastic differenti… ▽ More This paper is concerned with the problem of nonlinear filter stability of ergodic Markov processes. The main contribution is the conditional Poincaré inequality (PI), which is shown to yield filter stability. The proof is based upon a recently discovered duality which is used to transform the nonlinear filtering problem into a stochastic optimal control problem for a backward stochastic differential equation (BSDE). Based on these dual formalisms, a comparison is drawn between the stochastic stability of a Markov process and the filter stability. The latter relies on the conditional PI described in this paper, whereas the former relies on the standard form of PI. △ Less

Submitted 8 October, 2021; v1 submitted 26 March, 2021; originally announced March 2021.

Comments: 60th IEEE Conference on Decision and Control (CDC)

Optimal Transportation Methods in Nonlinear Filtering: The feedback particle filter

Authors: Amirhossein Taghvaei, Prashant G. Mehta

Abstract: Feedback particle filter (FPF) is a Monte-Carlo (MC) algorithm to approximate the solution of a stochastic filtering problem. In contrast to conventional particle filters, the Bayesian update step in FPF is implemented via a mean-field type feedback control law. The objective for this paper is to situate the development of FPF and related controlled interacting particle system algorithms within… ▽ More Feedback particle filter (FPF) is a Monte-Carlo (MC) algorithm to approximate the solution of a stochastic filtering problem. In contrast to conventional particle filters, the Bayesian update step in FPF is implemented via a mean-field type feedback control law. The objective for this paper is to situate the development of FPF and related controlled interacting particle system algorithms within the framework of optimal transportation theory. Starting from the simplest setting of the Bayes' update formula, a coupling viewpoint is introduced to construct particle filters. It is shown that the conventional importance sampling resampling particle filter implements an independent coupling. Design of optimal couplings is introduced first for the simple Gaussian settings and subsequently extended to derive the FPF algorithm. The final half of the paper provides a review of some of the salient aspects of the FPF algorithm including the feedback structure, algorithms for gain function design, and comparison with conventional particle filters. The comparison serves to illustrate the benefit of feedback in particle filtering. △ Less

Submitted 21 February, 2021; originally announced February 2021.

Minimum variance constrained estimator

Authors: Prabhat K. Mishra, Girish Chowdhary, Prashant G. Mehta

Abstract: This paper is concerned with the problem of state estimation for discrete-time linear systems in the presence of additional (equality or inequality) constraints on the state (or estimate). By use of the minimum variance duality, the estimation problem is converted into an optimal control problem. Two algorithmic solutions are described: the full information estimator (FIE) and the moving horizon e… ▽ More This paper is concerned with the problem of state estimation for discrete-time linear systems in the presence of additional (equality or inequality) constraints on the state (or estimate). By use of the minimum variance duality, the estimation problem is converted into an optimal control problem. Two algorithmic solutions are described: the full information estimator (FIE) and the moving horizon estimator (MHE). The main result is to show that the proposed estimator is stable in the sense of an observer. The proposed algorithm is distinct from the standard algorithm for constrained state estimation based upon the use of the minimum energy duality. The two are compared numerically on the benchmark batch reactor process model. △ Less

Submitted 7 December, 2021; v1 submitted 14 January, 2021; originally announced January 2021.

The Perturbative Resolvent Method: spectral densities of random matrix ensembles via perturbation theory

Authors: Wenping Cui, Jason W. Rocks, Pankaj Mehta

Abstract: We present a simple, perturbative approach for calculating spectral densities for random matrix ensembles in the thermodynamic limit we call the Perturbative Resolvent Method (PRM). The PRM is based on constructing a linear system of equations and calculating how the solutions to these equation change in response to a small perturbation using the zero-temperature cavity method. We illustrate the p… ▽ More We present a simple, perturbative approach for calculating spectral densities for random matrix ensembles in the thermodynamic limit we call the Perturbative Resolvent Method (PRM). The PRM is based on constructing a linear system of equations and calculating how the solutions to these equation change in response to a small perturbation using the zero-temperature cavity method. We illustrate the power of the method by providing simple analytic derivations of the Wigner Semi-circle Law for symmetric matrices, the Marchenko-Pastur Law for Wishart matrices, the spectral density for a product Wishart matrix composed of two square matrices, and the Circle and elliptic laws for real random matrices. △ Less

Submitted 1 December, 2020; originally announced December 2020.

Comments: 15 page, 4 figures

Optimality vs Stability Trade-off in Ensemble Kalman Filters

Authors: Amirhossein Taghvaei, Prashant G. Mehta, Tryphon T. Georgiou

Abstract: This paper is concerned with optimality and stability analysis of a family of ensemble Kalman filter (EnKF) algorithms. EnKF is commonly used as an alternative to the Kalman filter for high-dimensional problems, where storing the covariance matrix is computationally expensive. The algorithm consists of an ensemble of interacting particles driven by a feedback control law. The control law is design… ▽ More This paper is concerned with optimality and stability analysis of a family of ensemble Kalman filter (EnKF) algorithms. EnKF is commonly used as an alternative to the Kalman filter for high-dimensional problems, where storing the covariance matrix is computationally expensive. The algorithm consists of an ensemble of interacting particles driven by a feedback control law. The control law is designed such that, in the linear Gaussian setting and asymptotic limit of infinitely many particles, the mean and covariance of the particles follow the exact mean and covariance of the Kalman filter. The problem of finding a control law that is exact does not have a unique solution, reminiscent of the problem of finding a transport map between two distributions. A unique control law can be identified by introducing control cost functions, that are motivated by the optimal transportation problem or Schrödinger bridge problem. The objective of this paper is to study the relationship between optimality and long-term stability of a family of exact control laws. Remarkably, the control law that is optimal in the optimal transportation sense leads to an EnKF algorithm that is not stable. △ Less

Submitted 18 February, 2022; v1 submitted 19 October, 2020; originally announced October 2020.

Feedback Particle Filter for Collective Inference

Authors: Jin Won Kim, Amirhossein Taghvaei, Yongxin Chen, Prashant G. Mehta

Abstract: The purpose of this paper is to describe the feedback particle filter algorithm for problems where there are a large number ($M$) of non-interacting agents (targets) with a large number ($M$) of non-agent specific observations (measurements) that originate from these agents. In its basic form, the problem is characterized by data association uncertainty whereby the association between the observat… ▽ More The purpose of this paper is to describe the feedback particle filter algorithm for problems where there are a large number ($M$) of non-interacting agents (targets) with a large number ($M$) of non-agent specific observations (measurements) that originate from these agents. In its basic form, the problem is characterized by data association uncertainty whereby the association between the observations and agents must be deduced in addition to the agent state. In this paper, the large-$M$ limit is interpreted as a problem of collective inference. This viewpoint is used to derive the equation for the empirical distribution of the hidden agent states. A feedback particle filter (FPF) algorithm for this problem is presented and illustrated via numerical simulations. Results are presented for the Euclidean and the finite state-space cases, both in continuous-time settings. The classical FPF algorithm is shown to be the special case (with $M=1$) of these more general results. The simulations help show that the algorithm well approximates the empirical distribution of the hidden states for large $M$. △ Less

Submitted 17 February, 2021; v1 submitted 13 October, 2020; originally announced October 2020.

Comments: 15 pages, 2 figures. Submitted to FoDA

MSC Class: 60G35; 62M20 (Primary) 94A12 (Secondary)

Optimal Control of a Soft CyberOctopus Arm

Authors: Tixian Wang, Udit Halder, Heng-Sheng Chang, Mattia Gazzola, Prashant G. Mehta

Abstract: In this paper, we use the optimal control methodology to control a flexible, elastic Cosserat rod. An inspiration comes from stereotypical movement patterns in octopus arms, which are observed in a variety of manipulation tasks, such as reaching or fetching. To help uncover the mechanisms underlying these observed morphologies, we outline an optimal control-based framework. A single octopus arm is… ▽ More In this paper, we use the optimal control methodology to control a flexible, elastic Cosserat rod. An inspiration comes from stereotypical movement patterns in octopus arms, which are observed in a variety of manipulation tasks, such as reaching or fetching. To help uncover the mechanisms underlying these observed morphologies, we outline an optimal control-based framework. A single octopus arm is modeled as a Hamiltonian control system, where the continuum mechanics of the arm is modeled after the Cosserat rod theory, and internal, distributed muscle forces and couples are considered as controls. First order necessary optimality conditions are derived for an optimal control problem formulated for this infinite dimensional system. Solutions to this problem are obtained numerically by an iterative forward-backward algorithm. The state and adjoint equations are solved in a dynamic simulation environment, setting the stage for studying a broader class of optimal control problems. Trajectories that minimize control effort are demonstrated and qualitatively compared with observed behaviors. △ Less

Submitted 1 April, 2021; v1 submitted 2 October, 2020; originally announced October 2020.

Deep FPF: Gain function approximation in high-dimensional setting

Authors: S. Yagiz Olmez, Amirhossein Taghvaei, Prashant G. Mehta

Abstract: In this paper, we present a novel approach to approximate the gain function of the feedback particle filter (FPF). The exact gain function is the solution of a Poisson equation involving a probability-weighted Laplacian. The numerical problem is to approximate the exact gain function using only finitely many particles sampled from the probability distribution. Inspired by the recent success of t… ▽ More In this paper, we present a novel approach to approximate the gain function of the feedback particle filter (FPF). The exact gain function is the solution of a Poisson equation involving a probability-weighted Laplacian. The numerical problem is to approximate the exact gain function using only finitely many particles sampled from the probability distribution. Inspired by the recent success of the deep learning methods, we represent the gain function as a gradient of the output of a neural network. Thereupon considering a certain variational formulation of the Poisson equation, an optimization problem is posed for learning the weights of the neural network. A stochastic gradient algorithm is described for this purpose. The proposed approach has two significant properties/advantages: (i) The stochastic optimization algorithm allows one to process, in parallel, only a batch of samples (particles) ensuring good scaling properties with the number of particles; (ii) The remarkable representation power of neural networks means that the algorithm is potentially applicable and useful to solve high-dimensional problems. We numerically establish these two properties and provide extensive comparison to the existing approaches. △ Less

Submitted 2 October, 2020; originally announced October 2020.

Comments: To be presented at 59th IEEE Conference on Decision and Control, 2020

Convex Q-Learning, Part 1: Deterministic Optimal Control

Authors: Prashant G. Mehta, Sean P. Meyn

Abstract: It is well known that the extension of Watkins' algorithm to general function approximation settings is challenging: does the projected Bellman equation have a solution? If so, is the solution useful in the sense of generating a good policy? And, if the preceding questions are answered in the affirmative, is the algorithm consistent? These questions are unanswered even in the special case of Q-fun… ▽ More It is well known that the extension of Watkins' algorithm to general function approximation settings is challenging: does the projected Bellman equation have a solution? If so, is the solution useful in the sense of generating a good policy? And, if the preceding questions are answered in the affirmative, is the algorithm consistent? These questions are unanswered even in the special case of Q-function approximations that are linear in the parameter. The challenge seems paradoxical, given the long history of convex analytic approaches to dynamic programming. The paper begins with a brief survey of linear programming approaches to optimal control, leading to a particular over parameterization that lends itself to applications in reinforcement learning. The main conclusions are summarized as follows: (i) The new class of convex Q-learning algorithms is introduced based on the convex relaxation of the Bellman equation. Convergence is established under general conditions, including a linear function approximation for the Q-function. (ii) A batch implementation appears similar to the famed DQN algorithm (one engine behind AlphaZero). It is shown that in fact the algorithms are very different: while convex Q-learning solves a convex program that approximates the Bellman equation, theory for DQN is no stronger than for Watkins' algorithm with function approximation: (a) it is shown that both seek solutions to the same fixed point equation, and (b) the ODE approximations for the two algorithms coincide, and little is known about the stability of this ODE. These results are obtained for deterministic nonlinear systems with total cost criterion. Many extensions are proposed, including kernel implementation, and extension to MDP models. △ Less

Submitted 8 August, 2020; originally announced August 2020.

Comments: This pre-print is written in a tutorial style so it is accessible to new-comers. It will be a part of a handout for upcoming short courses on RL. A more compact version suitable for journal submission is in preparation

MSC Class: 68T05 (Primary) 93E35; 49L20 (Secondary)

On the Lyapunov Foster criterion and Poincaré inequality for Reversible Markov Chains

Authors: Amirhossein Taghvaei, Prashant G. Mehta

Abstract: This paper presents an elementary proof of stochastic stability of a discrete-time reversible Markov chain starting from a Foster-Lyapunov drift condition. Besides its relative simplicity, there are two salient features of the proof: (i) it relies entirely on functional-analytic non-probabilistic arguments; and (ii) it makes explicit the connection between a Foster-Lyapunov function and Poincaré i… ▽ More This paper presents an elementary proof of stochastic stability of a discrete-time reversible Markov chain starting from a Foster-Lyapunov drift condition. Besides its relative simplicity, there are two salient features of the proof: (i) it relies entirely on functional-analytic non-probabilistic arguments; and (ii) it makes explicit the connection between a Foster-Lyapunov function and Poincaré inequality. The proof is used to derive an explicit bound for the spectral gap. An extension to the non-reversible case is also presented. △ Less

Submitted 16 May, 2020; originally announced May 2020.

A Dual Characterization of Observability for Stochastic Systems

Authors: Jin W. Kim, Prashant G. Mehta

Abstract: This paper is concerned with a characterization of the observability for a continuous-time hidden Markov model where the state evolves as a general continuous-time Markov process and the observation process is modeled as nonlinear function of the state corrupted by the Gaussian measurement noise. The main technical tool is based on the recently discovered duality relationship between minimum varia… ▽ More This paper is concerned with a characterization of the observability for a continuous-time hidden Markov model where the state evolves as a general continuous-time Markov process and the observation process is modeled as nonlinear function of the state corrupted by the Gaussian measurement noise. The main technical tool is based on the recently discovered duality relationship between minimum variance estimation and stochastic optimal control: The observability is defined as a dual of the controllability for a certain backward stochastic differential equation. Based on the dual formulation, a test for observability is presented and related to literature. The proposed duality-based framework allows one to easily relate and compare the linear and the nonlinear systems. A side-by-side summary of this relationship is given in a tabular form (Table~1) △ Less

Submitted 21 February, 2020; v1 submitted 27 September, 2019; originally announced September 2019.

Comments: 7 pages, Revised to be submitted to 2020 MTNS Conference

An Optimal Control Derivation of Nonlinear Smoothing Equations

Authors: Jin W. Kim, Prashant G. Mehta

Abstract: The purpose of this paper is to review and highlight some connections between the problem of nonlinear smoothing and optimal control of the Liouville equation. The latter has been an active area of recent research interest owing to work in mean-field games and optimal transportation theory. The nonlinear smoothing problem is considered here for continuous-time Markov processes. The observation pro… ▽ More The purpose of this paper is to review and highlight some connections between the problem of nonlinear smoothing and optimal control of the Liouville equation. The latter has been an active area of recent research interest owing to work in mean-field games and optimal transportation theory. The nonlinear smoothing problem is considered here for continuous-time Markov processes. The observation process is modeled as a nonlinear function of a hidden state with an additive Gaussian measurement noise. A variational formulation is described based upon the relative entropy formula introduced by Newton and Mitter. The resulting optimal control problem is formulated on the space of probability distributions. The Hamilton's equation of the optimal control are related to the Zakai equation of nonlinear smoothing via the log transformation. The overall procedure is shown to generalize the classical Mortensen's minimum energy estimator for the linear Gaussian problem. △ Less

Submitted 22 March, 2023; v1 submitted 2 April, 2019; originally announced April 2019.

Journal ref: In: Advances in Dynamics, Optimization and Computation. SON 2020. Studies in Systems, Decision and Control, vol 304. Springer, pp. 295-311 (2020)

What is the Lagrangian for Nonlinear Filtering?

Authors: Jin W. Kim, Prashant G. Mehta, Sean P. Meyn

Abstract: Duality between estimation and optimal control is a problem of rich historical significance. The first duality principle appears in the seminal paper of Kalman-Bucy, where the problem of minimum variance estimation is shown to be dual to a linear quadratic (LQ) optimal control problem. Duality offers a constructive proof technique to derive the Kalman filter equation from the optimal control solut… ▽ More Duality between estimation and optimal control is a problem of rich historical significance. The first duality principle appears in the seminal paper of Kalman-Bucy, where the problem of minimum variance estimation is shown to be dual to a linear quadratic (LQ) optimal control problem. Duality offers a constructive proof technique to derive the Kalman filter equation from the optimal control solution. This paper generalizes the classical duality result of Kalman-Bucy to the nonlinear filter: The state evolves as a continuous-time Markov process and the observation is a nonlinear function of state corrupted by an additive Gaussian noise. A dual process is introduced as a backward stochastic differential equation (BSDE). The process is used to transform the problem of minimum variance estimation into an optimal control problem. Its solution is obtained from an application of the maximum principle, and subsequently used to derive the equation of the nonlinear filter. The classical duality result of Kalman-Bucy is shown to be a special case. △ Less

Submitted 24 October, 2019; v1 submitted 26 March, 2019; originally announced March 2019.

Comments: 8 pages, 58th IEEE Conference on Decision and Control (Dec. 2019)

Optimal Rate of Convergence for Quasi-Stochastic Approximation

Authors: Andrey Bernstein, Yue Chen, Marcello Colombino, Emiliano Dall'Anese, Prashant Mehta, Sean Meyn

Abstract: The Robbins-Monro stochastic approximation algorithm is a foundation of many algorithmic frameworks for reinforcement learning (RL), and often an efficient approach to solving (or approximating the solution to) complex optimal control problems. However, in many cases practitioners are unable to apply these techniques because of an inherent high variance. This paper aims to provide a general founda… ▽ More The Robbins-Monro stochastic approximation algorithm is a foundation of many algorithmic frameworks for reinforcement learning (RL), and often an efficient approach to solving (or approximating the solution to) complex optimal control problems. However, in many cases practitioners are unable to apply these techniques because of an inherent high variance. This paper aims to provide a general foundation for "quasi-stochastic approximation," in which all of the processes under consideration are deterministic, much like quasi-Monte-Carlo for variance reduction in simulation. The variance reduction can be substantial, subject to tuning of pertinent parameters in the algorithm. This paper introduces a new coupling argument to establish optimal rate of convergence provided the gain is sufficiently large. These results are established for linear models, and tested also in non-ideal settings. A major application of these general results is a new class of RL algorithms for deterministic state space models. In this setting, the main contribution is a class of algorithms for approximating the value function for a given policy, using a different policy designed to introduce exploration. △ Less

Submitted 17 March, 2019; originally announced March 2019.

Diffusion map-based algorithm for Gain function approximation in the Feedback Particle Filter

Authors: Amirhossein Taghvaei, Prashant G. Mehta, Sean P. Meyn

Abstract: Feedback particle filter (FPF) is a numerical algorithm to approximate the solution of the nonlinear filtering problem in continuous-time settings. In any numerical implementation of the FPF algorithm, the main challenge is to numerically approximate the so-called gain function. A numerical algorithm for gain function approximation is the subject of this paper. The exact gain function is the solut… ▽ More Feedback particle filter (FPF) is a numerical algorithm to approximate the solution of the nonlinear filtering problem in continuous-time settings. In any numerical implementation of the FPF algorithm, the main challenge is to numerically approximate the so-called gain function. A numerical algorithm for gain function approximation is the subject of this paper. The exact gain function is the solution of a Poisson equation involving a probability-weighted Laplacian $Δ_ρ$. The numerical problem is to approximate this solution using {\em only} finitely many particles sampled from the probability distribution $ρ$. A diffusion map-based algorithm was proposed by the authors in a prior work to solve this problem. The algorithm is named as such because it involves, as an intermediate step, a diffusion map approximation of the exact semigroup $e^{Δ_ρ}$. The original contribution of this paper is to carry out a rigorous error analysis of the diffusion map-based algorithm. The error is shown to include two components: bias and variance. The bias results from the diffusion map approximation of the exact semigroup. The variance arises because of finite sample size. Scalings and upper bounds are derived for bias and variance. These bounds are then illustrated with numerical experiments that serve to emphasize the effects of problem dimension and sample size. The proposed algorithm is applied to two filtering examples and comparisons provided with the sequential importance resampling (SIR) particle filter. △ Less

Submitted 30 September, 2019; v1 submitted 19 February, 2019; originally announced February 2019.

Accelerated Flow for Probability Distributions

Authors: Amirhossein Taghvaei, Prashant G. Mehta

Abstract: This paper presents a methodology and numerical algorithms for constructing accelerated gradient flows on the space of probability distributions. In particular, we extend the recent variational formulation of accelerated gradient methods in (wibisono, et. al. 2016) from vector valued variables to probability distributions. The variational problem is modeled as a mean-field optimal control problem.… ▽ More This paper presents a methodology and numerical algorithms for constructing accelerated gradient flows on the space of probability distributions. In particular, we extend the recent variational formulation of accelerated gradient methods in (wibisono, et. al. 2016) from vector valued variables to probability distributions. The variational problem is modeled as a mean-field optimal control problem. The maximum principle of optimal control theory is used to derive Hamilton's equations for the optimal gradient flow. The Hamilton's equation are shown to achieve the accelerated form of density transport from any initial probability distribution to a target probability distribution. A quantitative estimate on the asymptotic convergence rate is provided based on a Lyapunov function construction, when the objective functional is displacement convex. Two numerical approximations are presented to implement the Hamilton's equations as a system of $N$ interacting particles. The continuous limit of the Nesterov's algorithm is shown to be a special case with $N=1$. The algorithm is illustrated with numerical examples. △ Less

Submitted 10 January, 2019; v1 submitted 10 January, 2019; originally announced January 2019.

An Approach to Duality in Nonlinear Filtering

Authors: Jin W. Kim, Amirhossein Taghvaei, Prashant G. Mehta, Sean P. Meyn

Abstract: This paper revisits the question of duality between minimum variance estimation and optimal control first described for the linear Gaussian case in the celebrated paper of Kalman and Bucy. A duality result is established for nonlinear filtering, mirroring closely the original Kalman-Bucy duality of control and estimation for linear systems. The result for the finite state-space continuous time Mar… ▽ More This paper revisits the question of duality between minimum variance estimation and optimal control first described for the linear Gaussian case in the celebrated paper of Kalman and Bucy. A duality result is established for nonlinear filtering, mirroring closely the original Kalman-Bucy duality of control and estimation for linear systems. The result for the finite state-space continuous time Markov chain is presented. It's solution is used to derive the classical Wonham filter. △ Less

Submitted 26 March, 2019; v1 submitted 27 September, 2018; originally announced September 2018.

Comments: 6 pages, 2019 American Control Conference

Error Analysis of the Stochastic Linear Feedback Particle Filter

Authors: Amirhossein Taghvaei, Prashant G. Mehta

Abstract: This paper is concerned with the convergence and long-term stability analysis of the feedback particle filter (FPF) algorithm. The FPF is an interacting system of $N$ particles where the interaction is designed such that the empirical distribution of the particles approximates the posterior distribution. It is known that in the mean-field limit ($N=\infty$), the distribution of the particles is eq… ▽ More This paper is concerned with the convergence and long-term stability analysis of the feedback particle filter (FPF) algorithm. The FPF is an interacting system of $N$ particles where the interaction is designed such that the empirical distribution of the particles approximates the posterior distribution. It is known that in the mean-field limit ($N=\infty$), the distribution of the particles is equal to the posterior distribution. However little is known about the convergence to the mean-field limit. In this paper, we consider the FPF algorithm for the linear Gaussian setting. In this setting, the algorithm is similar to the ensemble Kalman-Bucy filter algorithm. Although these algorithms have been numerically evaluated and widely used in applications, their convergence and long-term stability analysis remains an active area of research. In this paper, we show that, (i) the mean-field limit is well-defined with a unique strong solution; (ii) the mean-field process is stable with respect to the initial condition; (iii) we provide conditions such that the finite-$N$ system is long term stable and we obtain some mean-squared error estimates that are uniform in time. △ Less

Submitted 20 September, 2018; originally announced September 2018.

Constrained optimization as ecological dynamics with applications to random quadratic programming in high dimensions

Authors: Pankaj Mehta, Wenping Cui, Ching-Hao Wang, Robert Marsland III

Abstract: Quadratic programming (QP) is a common and important constrained optimization problem. Here, we derive a surprising duality between constrained optimization with inequality constraints -- of which QP is a special case -- and consumer resource models describing ecological dynamics. Combining this duality with a recent `cavity solution', we analyze high-dimensional, random QP where the optimization… ▽ More Quadratic programming (QP) is a common and important constrained optimization problem. Here, we derive a surprising duality between constrained optimization with inequality constraints -- of which QP is a special case -- and consumer resource models describing ecological dynamics. Combining this duality with a recent `cavity solution', we analyze high-dimensional, random QP where the optimization function and constraints are drawn randomly. Our theory shows remarkable agreement with numerics and points to a deep connection between optimization, dynamical systems, and ecology. △ Less

Submitted 11 September, 2018; originally announced September 2018.

Comments: 12 pages, 3 figures with appendix

Journal ref: Phys. Rev. E 99, 052111 (2019)

Data-driven framework for real-time thermospheric density estimation

Authors: Piyush M. Mehta, Richard Linares

Abstract: In this paper, we demonstrate a new data-driven framework for real-time neutral density estimation via model-data fusion in quasi-physical ionosphere-thermosphere models. The framework has two main components: (i) the development of a quasi-physical dynamic reduced order model (ROM) that uses a linear approximation of the underlying dynamics and effect of the drivers, and (ii) dynamic calibration… ▽ More In this paper, we demonstrate a new data-driven framework for real-time neutral density estimation via model-data fusion in quasi-physical ionosphere-thermosphere models. The framework has two main components: (i) the development of a quasi-physical dynamic reduced order model (ROM) that uses a linear approximation of the underlying dynamics and effect of the drivers, and (ii) dynamic calibration of the ROM through estimation of the ROM coefficients that represent the model parameters. We have previously demonstrated the development of a quasi-physical ROM using simulation output from a physical model and assimilation of non-operational density estimates derived from accelerometer measurements along a single orbit. In this paper, we demonstrate the potential of the framework for use with operational measurements. We use simulated GPS-derived orbit ephemerides with 5 minute resolution as measurements. The framework is a first of its kind, simple yet robust and accurate method with high potential for providing real-time operational updates to the state of the upper atmosphere using quasi-physical models with inherent forecasting/predictive capabilities. △ Less

Submitted 17 August, 2018; originally announced August 2018.

Derivation and Extensions of the Linear Feedback Particle Filter based on Duality Formalisms

Authors: Jin W. Kim, Amirhossein Taghvaei, Prashant G. Mehta

Abstract: This paper is concerned with a duality-based approach to derive the linear feedback particle filter (FPF). The FPF is a controlled interacting particle system where the control law is designed to provide an exact solution for the nonlinear filtering problem. For the linear Gaussian special case, certain simplifications arise whereby the linear FPF is identical to the square-root form of the ensemb… ▽ More This paper is concerned with a duality-based approach to derive the linear feedback particle filter (FPF). The FPF is a controlled interacting particle system where the control law is designed to provide an exact solution for the nonlinear filtering problem. For the linear Gaussian special case, certain simplifications arise whereby the linear FPF is identical to the square-root form of the ensemble Kalman filter. For this and for the more general nonlinear non-Gaussian case, it has been an open problem to derive/interpret the FPF control law as a solution of an optimal control problem. In this paper, certain duality-based arguments are employed to transform the filtering problem into an optimal control problem. Its solution is shown to yield the deterministic form of the linear FPF. An extension is described to incorporate stochastic effects due to noise leading to a novel homotopy of exact ensemble Kalman filters. All the derivations are based on duality formalisms. △ Less

Submitted 11 April, 2018; originally announced April 2018.

Glassy Phase of Optimal Quantum Control

Authors: Alexandre G. R. Day, Marin Bukov, Phillip Weinberg, Pankaj Mehta, Dries Sels

Abstract: We study the problem of preparing a quantum many-body system from an initial to a target state by optimizing the fidelity over the family of bang-bang protocols. We present compelling numerical evidence for a universal spin-glass-like transition controlled by the protocol time duration. The glassy critical point is marked by a proliferation of protocols with close-to-optimal fidelity and with a tr… ▽ More We study the problem of preparing a quantum many-body system from an initial to a target state by optimizing the fidelity over the family of bang-bang protocols. We present compelling numerical evidence for a universal spin-glass-like transition controlled by the protocol time duration. The glassy critical point is marked by a proliferation of protocols with close-to-optimal fidelity and with a true optimum that appears exponentially difficult to locate. Using a machine learning (ML) inspired framework based on the manifold learning algorithm t-SNE, we are able to visualize the geometry of the high-dimensional control landscape in an effective low-dimensional representation. Across the transition, the control landscape features an exponential number of clusters separated by extensive barriers, which bears a strong resemblance with replica symmetry breaking in spin glasses and random satisfiability problems. We further show that the quantum control landscape maps onto a disorder-free classical Ising model with frustrated nonlocal, multibody interactions. Our work highlights an intricate but unexpected connection between optimal quantum control and spin glass physics, and shows how tools from ML can be used to visualize and understand glassy optimization landscapes. △ Less

Submitted 8 February, 2019; v1 submitted 28 March, 2018; originally announced March 2018.

Comments: Modified figures in appendix and main text (color schemes). Corrected references. Added figures in SI and pseudo-code

Journal ref: Phys. Rev. Lett. 122, 020601 (2019)

Error Analysis for the Linear Feedback Particle Filter

Authors: Amirhossein Taghvaei, Prashant G. Mehta

Abstract: This paper is concerned with the convergence and the error analysis for the feedback particle filter (FPF) algorithm. The FPF is a controlled interacting particle system where the control law is designed to solve the nonlinear filtering problem. For the linear Gaussian case, certain simplifications arise whereby the linear FPF reduces to one form of the ensemble Kalman filter. For this and for the… ▽ More This paper is concerned with the convergence and the error analysis for the feedback particle filter (FPF) algorithm. The FPF is a controlled interacting particle system where the control law is designed to solve the nonlinear filtering problem. For the linear Gaussian case, certain simplifications arise whereby the linear FPF reduces to one form of the ensemble Kalman filter. For this and for the more general nonlinear non-Gaussian case, it has been an open problem to relate the convergence and error properties of the finite-N algorithm to the mean-field limit (where the exactness results have been obtained). In this paper, the equations for empirical mean and covariance are derived for the finite-N linear FPF. Remarkably, for a certain deterministic form of FPF, the equations for mean and variance are identical to the Kalman filter. This allows strong conclusions on convergence and error properties based on the classical filter stability theory for the Kalman filter. It is shown that the error converges to zero even with finite number of particles. The paper also presents propagation of chaos estimates for the finite-N linear filter. The error estimates are illustrated with numerical experiments. △ Less

Submitted 30 October, 2017; originally announced October 2017.

How regularization affects the critical points in linear networks

Authors: Amirhossein Taghvaei, Jin W. Kim, Prashant G. Mehta

Abstract: This paper is concerned with the problem of representing and learning a linear transformation using a linear neural network. In recent years, there has been a growing interest in the study of such networks in part due to the successes of deep learning. The main question of this body of research and also of this paper pertains to the existence and optimality properties of the critical points of the… ▽ More This paper is concerned with the problem of representing and learning a linear transformation using a linear neural network. In recent years, there has been a growing interest in the study of such networks in part due to the successes of deep learning. The main question of this body of research and also of this paper pertains to the existence and optimality properties of the critical points of the mean-squared loss function. The primary concern here is the robustness of the critical points with regularization of the loss function. An optimal control model is introduced for this purpose and a learning algorithm (regularized form of backprop) derived for the same using the Hamilton's formulation of optimal control. The formulation is used to provide a complete characterization of the critical points in terms of the solutions of a nonlinear matrix-valued equation, referred to as the characteristic equation. Analytical and numerical tools from bifurcation theory are used to compute the critical points via the solutions of the characteristic equation. The main conclusion is that the critical point diagram can be fundamentally different even with arbitrary small amounts of regularization. △ Less

Submitted 27 September, 2017; originally announced September 2017.

Kalman Filter and its Modern Extensions for the Continuous-time Nonlinear Filtering Problem

Authors: Amirhossein Taghvaei, Jana de Wiljes, Prashant G. Mehta, Sebastian Reich

Abstract: This paper is concerned with the filtering problem in continuous-time. Three algorithmic solution approaches for this problem are reviewed: (i) the classical Kalman-Bucy filter which provides an exact solution for the linear Gaussian problem, (ii) the ensemble Kalman-Bucy filter (EnKBF) which is an approximate filter and represents an extension of the Kalman-Bucy filter to nonlinear problems, and… ▽ More This paper is concerned with the filtering problem in continuous-time. Three algorithmic solution approaches for this problem are reviewed: (i) the classical Kalman-Bucy filter which provides an exact solution for the linear Gaussian problem, (ii) the ensemble Kalman-Bucy filter (EnKBF) which is an approximate filter and represents an extension of the Kalman-Bucy filter to nonlinear problems, and (iii) the feedback particle filter (FPF) which represents an extension of the EnKBF and furthermore provides for an consistent solution in the general nonlinear, non-Gaussian case. The common feature of the three algorithms is the gain times error formula to implement the update step (to account for conditioning due to the observations) in the filter. In contrast to the commonly used sequential Monte Carlo methods, the EnKBF and FPF avoid the resampling of the particles in the importance sampling update step. Moreover, the feedback control structure provides for error correction potentially leading to smaller simulation variance and improved stability properties. The paper also discusses the issue of non-uniqueness of the filter update formula and formulates a novel approximation algorithm based on ideas from optimal transport and coupling of measures. Performance of this and other algorithms is illustrated for a numerical example. △ Less

Submitted 21 December, 2017; v1 submitted 21 February, 2017; originally announced February 2017.

Feedback Particle Filter on Matrix Lie Groups

Authors: Chi Zhang, Amirhossein Taghvaei, Prashant G. Mehta

Abstract: This paper is concerned with the problem of continuous-time nonlinear filtering for stochastic processes on a connected matrix Lie group. The main contribution of this paper is to derive the feedback particle filter (FPF) algorithm for this problem. In its general form, the FPF is shown to provide a coordinate-free description of the filter that automatically satisfies the geometric constraints of… ▽ More This paper is concerned with the problem of continuous-time nonlinear filtering for stochastic processes on a connected matrix Lie group. The main contribution of this paper is to derive the feedback particle filter (FPF) algorithm for this problem. In its general form, the FPF is shown to provide a coordinate-free description of the filter that automatically satisfies the geometric constraints of the manifold. The particle dynamics are encapsulated in a Stratonovich stochastic differential equation that retains the feedback structure of the original (Euclidean) FPF. The implementation of the filter requires a solution of a Poisson equation on the Lie group, and two numerical algorithms are described for this purpose. As an example, the FPF is applied to the problem of attitude estimation - a nonlinear filtering problem on the Lie group SO(3). The formulae of the filter are described using both the rotation matrix and the quaternion coordinates. Comparisons are also provided between the FPF and some popular algorithms for attitude estimation, namely the multiplicative EKF, the unscented quaternion estimator, the left invariant EKF, and the invariant ensemble Kalman filter. Numerical simulations are presented to illustrate the comparisons. △ Less

Submitted 9 January, 2017; originally announced January 2017.

Comments: 33 pages

A Controlled Particle Filter for Global Optimization

Authors: Chi Zhang, Amirhossein Taghvaei, Prashant G. Mehta

Abstract: A particle filter is introduced to numerically approximate a solution of the global optimization problem. The theoretical significance of this work comes from its variational aspects: (i) the proposed particle filter is a controlled interacting particle system where the control input represents the solution of a mean-field type optimal control problem; and (ii) the associated density transport is… ▽ More A particle filter is introduced to numerically approximate a solution of the global optimization problem. The theoretical significance of this work comes from its variational aspects: (i) the proposed particle filter is a controlled interacting particle system where the control input represents the solution of a mean-field type optimal control problem; and (ii) the associated density transport is shown to be a gradient flow (steepest descent) for the optimal value function, with respect to the Kullback--Leibler divergence. The optimal control construction of the particle filter is a significant departure from the classical importance sampling-resampling based approaches. There are several practical advantages: (i) resampling, reproduction, death or birth of particles is avoided; (ii) simulation variance can potentially be reduced by applying feedback control principles; and (iii) the parametric approximation naturally arises as a special case. The latter also suggests systematic approaches for numerical approximation of the optimal control law. The theoretical results are illustrated with numerical examples. △ Less

Submitted 9 January, 2017; originally announced January 2017.

Comments: 33 pages

Error Estimates for the Kernel Gain Function Approximation in the Feedback Particle Filter

Authors: Amirhossein Taghvaei, Prashant G. Mehta, Sean P. Meyn

Abstract: This paper is concerned with the analysis of the kernel-based algorithm for gain function approximation in the feedback particle filter. The exact gain function is the solution of a Poisson equation involving a probability-weighted Laplacian. The kernel-based method -- introduced in our prior work -- allows one to approximate this solution using {\em only} particles sampled from the probability di… ▽ More This paper is concerned with the analysis of the kernel-based algorithm for gain function approximation in the feedback particle filter. The exact gain function is the solution of a Poisson equation involving a probability-weighted Laplacian. The kernel-based method -- introduced in our prior work -- allows one to approximate this solution using {\em only} particles sampled from the probability distribution. This paper describes new representations and algorithms based on the kernel-based method. Theory surrounding the approximation is improved and a novel formula for the gain function approximation is derived. A procedure for carrying out error analysis of the approximation is introduced. Certain asymptotic estimates for bias and variance are derived for the general nonlinear non-Gaussian case. Comparison with the constant gain function approximation is provided. The results are illustrated with the aid of some numerical experiments. △ Less

Submitted 16 December, 2016; originally announced December 2016.

Attitude Estimation with Feedback Particle Filter

Authors: Chi Zhang, Amirhossein Taghvaei, Prashant G. Mehta

Abstract: This paper presents theory, application, and comparisons of the feedback particle filter (FPF) algorithm for the problem of attitude estimation. The paper builds upon our recent work on the exact FPF solution of the continuous-time nonlinear filtering problem on compact Lie groups. In this paper, the details of the FPF algorithm are presented for the problem of attitude estimation - a nonlinear fi… ▽ More This paper presents theory, application, and comparisons of the feedback particle filter (FPF) algorithm for the problem of attitude estimation. The paper builds upon our recent work on the exact FPF solution of the continuous-time nonlinear filtering problem on compact Lie groups. In this paper, the details of the FPF algorithm are presented for the problem of attitude estimation - a nonlinear filtering problem on SO(3). The quaternions are employed for computational purposes. The algorithm requires a numerical solution of the filter gain function, and two methods are applied for this purpose. Comparisons are also provided between the FPF and some popular algorithms for attitude estimation on SO(3), including the invariant EKF, the multiplicative EKF, and the unscented Kalman filter. Simulation results are presented that help illustrate the comparisons. △ Less

Submitted 5 April, 2016; originally announced April 2016.

Comments: 8 pages, 2 figures

Gain Function Approximation in the Feedback Particle Filter

Authors: Amirhossein Taghvaei, Prashant G. Mehta

Abstract: This paper is concerned with numerical algorithms for gain function approximation in the feedback particle filter. The exact gain function is the solution of a Poisson equation involving a probability-weighted Laplacian. The problem is to approximate this solution using only particles sampled from the probability distribution. Two algorithms are presented: a Galerkin algorithm and a kernel-based a… ▽ More This paper is concerned with numerical algorithms for gain function approximation in the feedback particle filter. The exact gain function is the solution of a Poisson equation involving a probability-weighted Laplacian. The problem is to approximate this solution using only particles sampled from the probability distribution. Two algorithms are presented: a Galerkin algorithm and a kernel-based algorithm. Both the algorithms are adapted to the samples and do not require approximation of the probability distribution as an intermediate step. The paper contains error analysis for the algorithms as well as some comparative numerical results for a non-Gaussian distribution. These algorithms are also applied and illustrated for a simple nonlinear filtering example. △ Less

Submitted 17 March, 2016; originally announced March 2016.

An Optimal Transport Formulation of the Linear Feedback Particle Filter

Authors: Amirhossein Taghvaei, Prashant G. Mehta

Abstract: Feedback particle filter (FPF) is an algorithm to numerically approximate the solution of the nonlinear filtering problem in continuous time. The algorithm implements a feedback control law for a system of particles such that the empirical distribution of particles approximates the posterior distribution. However, it has been noted in the literature that the feedback control law is not unique. To… ▽ More Feedback particle filter (FPF) is an algorithm to numerically approximate the solution of the nonlinear filtering problem in continuous time. The algorithm implements a feedback control law for a system of particles such that the empirical distribution of particles approximates the posterior distribution. However, it has been noted in the literature that the feedback control law is not unique. To find a unique control law, the filtering task is formulated here as an optimal transportation problem between the prior and the posterior distributions. Based on this formulation, a time stepping optimization procedure is proposed for the optimal control design. A key difference between the optimal control law and the one in the original FPF, is the replacement of noise term with a deterministic term. This difference serves to decreases the simulation variance, as illustrated with a simple numerical example. △ Less

Submitted 7 October, 2015; originally announced October 2015.

Feedback Particle Filter on Matrix Lie Groups

Authors: Chi Zhang, Amirhossein Taghvaei, Prashant G. Mehta

Abstract: This paper is concerned with the problem of continuous-time nonlinear filtering for stochastic processes on a compact and connected matrix Lie group without boundary, e.g. SO(n) and SE(n), in the presence of real-valued observations. This problem is important to numerous applications in attitude estimation, visual tracking and robotic localization. The main contribution of this paper is to derive… ▽ More This paper is concerned with the problem of continuous-time nonlinear filtering for stochastic processes on a compact and connected matrix Lie group without boundary, e.g. SO(n) and SE(n), in the presence of real-valued observations. This problem is important to numerous applications in attitude estimation, visual tracking and robotic localization. The main contribution of this paper is to derive the feedback particle filter (FPF) algorithm for this problem. In its general form, the FPF provides a coordinate-free description of the filter that furthermore satisfies the geometric constraints of the manifold. The particle dynamics are encapsulated in a Stratonovich stochastic differential equation that preserves the feedback structure of the original Euclidean FPF. Specific examples for SO(2) and SO(3) are provided to help illustrate the filter using the phase and the quaternion coordinates, respectively. △ Less

Submitted 5 October, 2015; originally announced October 2015.

Comments: 7 pages, submitted for 2016 American Control Conference