-
Robust Model Based Reinforcement Learning Using $\mathcal{L}_1$ Adaptive Control
Authors:
Minjun Sung,
Sambhu H. Karumanchi,
Aditya Gahlawat,
Naira Hovakimyan
Abstract:
We introduce $\mathcal{L}_1$-MBRL, a control-theoretic augmentation scheme for Model-Based Reinforcement Learning (MBRL) algorithms. Unlike model-free approaches, MBRL algorithms learn a model of the transition function using data and use it to design a control input. Our approach generates a series of approximate control-affine models of the learned transition function according to the proposed s…
▽ More
We introduce $\mathcal{L}_1$-MBRL, a control-theoretic augmentation scheme for Model-Based Reinforcement Learning (MBRL) algorithms. Unlike model-free approaches, MBRL algorithms learn a model of the transition function using data and use it to design a control input. Our approach generates a series of approximate control-affine models of the learned transition function according to the proposed switching law. Using the approximate model, control input produced by the underlying MBRL is perturbed by the $\mathcal{L}_1$ adaptive control, which is designed to enhance the robustness of the system against uncertainties. Importantly, this approach is agnostic to the choice of MBRL algorithm, enabling the use of the scheme with various MBRL algorithms. MBRL algorithms with $\mathcal{L}_1$ augmentation exhibit enhanced performance and sample efficiency across multiple MuJoCo environments, outperforming the original MBRL algorithms, both with and without system noise.
△ Less
Submitted 21 March, 2024;
originally announced March 2024.
-
$\mathcal{L}_1$Quad: $\mathcal{L}_1$ Adaptive Augmentation of Geometric Control for Agile Quadrotors with Performance Guarantees
Authors:
Zhuohuan Wu,
Sheng Cheng,
Pan Zhao,
Aditya Gahlawat,
Kasey A. Ackerman,
Arun Lakshmanan,
Chengyu Yang,
Jiahao Yu,
Naira Hovakimyan
Abstract:
Quadrotors that can operate predictably in the presence of imperfect model knowledge and external disturbances are crucial in safety-critical applications. We present L1Quad, a control architecture that ensures uniformly bounded transient response of the quadrotor's uncertain dynamics on the special Euclidean group SE(3). By leveraging the geometric controller and the L1 adaptive controller, the L…
▽ More
Quadrotors that can operate predictably in the presence of imperfect model knowledge and external disturbances are crucial in safety-critical applications. We present L1Quad, a control architecture that ensures uniformly bounded transient response of the quadrotor's uncertain dynamics on the special Euclidean group SE(3). By leveraging the geometric controller and the L1 adaptive controller, the L1Quad architecture provides a theoretically justified framework for the design and analysis of quadrotor's tracking controller in the presence of nonlinear (time- and state-dependent) uncertainties on both the translational and rotational dynamics. In addition, we validate the performance of the L1Quad architecture through extensive experiments for eleven types of uncertainties across various trajectories. The results demonstrate that the L1Quad can achieve consistently small tracking errors despite the uncertainties and disturbances and significantly outperforms existing state-of-the-art controllers.
△ Less
Submitted 19 December, 2024; v1 submitted 14 February, 2023;
originally announced February 2023.
-
Guaranteed Trajectory Tracking under Learned Dynamics with Contraction Metrics and Disturbance Estimation
Authors:
Pan Zhao,
Ziyao Guo,
Yikun Cheng,
Aditya Gahlawat,
Hyungsoo Kang,
Naira Hovakimyan
Abstract:
This paper presents an approach to trajectory-centric learning control based on contraction metrics and disturbance estimation for nonlinear systems subject to matched uncertainties. The approach uses deep neural networks to learn uncertain dynamics while still providing guarantees of transient tracking performance throughout the learning phase. Within the proposed approach, a disturbance estimati…
▽ More
This paper presents an approach to trajectory-centric learning control based on contraction metrics and disturbance estimation for nonlinear systems subject to matched uncertainties. The approach uses deep neural networks to learn uncertain dynamics while still providing guarantees of transient tracking performance throughout the learning phase. Within the proposed approach, a disturbance estimation law is adopted to estimate the pointwise value of the uncertainty, with pre-computable estimation error bounds (EEBs). The learned dynamics, the estimated disturbances, and the EEBs are then incorporated in a robust Riemann energy condition to compute the control law that guarantees exponential convergence of actual trajectories to desired ones throughout the learning phase, even when the learned model is poor. On the other hand, with improved accuracy, the learned model can help improve the robustness of the tracking controller, e.g., against input delays, and can be incorporated to plan better trajectories with improved performance, e.g., lower energy consumption and shorter travel time.The proposed framework is validated on a planar quadrotor example.
△ Less
Submitted 23 July, 2024; v1 submitted 15 December, 2021;
originally announced December 2021.
-
Safe Sampling-Based Air-Ground Rendezvous Algorithm for Complex Urban Environments
Authors:
Gabriel Barsi Haberfeld,
Aditya Gahlawat,
Naira Hovakimyan
Abstract:
Demand for fast and economical parcel deliveries in urban environments has risen considerably in recent years. A framework envisions efficient last-mile delivery in urban environments by leveraging a network of ride-sharing vehicles, where Unmanned Aerial Systems (UASs) drop packages on said vehicles, which then cover the majority of the distance before final aerial delivery. Notably, we consider…
▽ More
Demand for fast and economical parcel deliveries in urban environments has risen considerably in recent years. A framework envisions efficient last-mile delivery in urban environments by leveraging a network of ride-sharing vehicles, where Unmanned Aerial Systems (UASs) drop packages on said vehicles, which then cover the majority of the distance before final aerial delivery. Notably, we consider the problem of planning a rendezvous path for the UAS to reach a human driver, who may choose between N possible paths and has uncertain behavior, while meeting strict safety constraints. The long planning horizon and safety constraints require robust heuristics that combine learning and optimal control using Gaussian Process Regression, sampling-based optimization, and Model Predictive Control. The resulting algorithm is computationally efficient and shown to be effective in a variety of qualitative scenarios.
△ Less
Submitted 12 March, 2021;
originally announced March 2021.
-
Distributed Algorithms for Linearly-Solvable Optimal Control in Networked Multi-Agent Systems
Authors:
Neng Wan,
Aditya Gahlawat,
Naira Hovakimyan,
Evangelos A. Theodorou,
Petros G. Voulgaris
Abstract:
Distributed algorithms for both discrete-time and continuous-time linearly solvable optimal control (LSOC) problems of networked multi-agent systems (MASs) are investigated in this paper. A distributed framework is proposed to partition the optimal control problem of a networked MAS into several local optimal control problems in factorial subsystems, such that each (central) agent behaves optimall…
▽ More
Distributed algorithms for both discrete-time and continuous-time linearly solvable optimal control (LSOC) problems of networked multi-agent systems (MASs) are investigated in this paper. A distributed framework is proposed to partition the optimal control problem of a networked MAS into several local optimal control problems in factorial subsystems, such that each (central) agent behaves optimally to minimize the joint cost function of a subsystem that comprises a central agent and its neighboring agents, and the local control actions (policies) only rely on the knowledge of local observations. Under this framework, we not only preserve the correlations between neighboring agents, but moderate the communication and computational complexities by decentralizing the sampling and computational processes over the network. For discrete-time systems modeled by Markov decision processes, the joint Bellman equation of each subsystem is transformed into a system of linear equations and solved using parallel programming. For continuous-time systems modeled by Itô diffusion processes, the joint optimality equation of each subsystem is converted into a linear partial differential equation, whose solution is approximated by a path integral formulation and a sample-efficient relative entropy policy search algorithm, respectively. The learned control policies are generalized to solve the unlearned tasks by resorting to the compositionality principle, and illustrative examples of cooperative UAV teams are provided to verify the effectiveness and advantages of these algorithms.
△ Less
Submitted 17 February, 2021;
originally announced February 2021.
-
Cooperative Path Integral Control for Stochastic Multi-Agent Systems
Authors:
Neng Wan,
Aditya Gahlawat,
Naira Hovakimyan,
Evangelos A. Theodorou,
Petros G. Voulgaris
Abstract:
A distributed stochastic optimal control solution is presented for cooperative multi-agent systems. The network of agents is partitioned into multiple factorial subsystems, each of which consists of a central agent and neighboring agents. Local control actions that rely only on agents' local observations are designed to optimize the joint cost functions of subsystems. When solving for the local co…
▽ More
A distributed stochastic optimal control solution is presented for cooperative multi-agent systems. The network of agents is partitioned into multiple factorial subsystems, each of which consists of a central agent and neighboring agents. Local control actions that rely only on agents' local observations are designed to optimize the joint cost functions of subsystems. When solving for the local control actions, the joint optimality equation for each subsystem is cast as a linear partial differential equation and solved using the Feynman-Kac formula. The solution and the optimal control action are then formulated as path integrals and approximated by a Monte-Carlo method. Numerical verification is provided through a simulation example consisting of a team of cooperative UAVs.
△ Less
Submitted 20 March, 2021; v1 submitted 30 September, 2020;
originally announced September 2020.
-
Compositionality of Linearly Solvable Optimal Control in Networked Multi-Agent Systems
Authors:
Lin Song,
Neng Wan,
Aditya Gahlawat,
Naira Hovakimyan,
Evangelos A. Theodorou
Abstract:
In this paper, we discuss the methodology of generalizing the optimal control law from learned component tasks to unlearned composite tasks on Multi-Agent Systems (MASs), by using the linearity composition principle of linearly solvable optimal control (LSOC) problems. The proposed approach achieves both the compositionality and optimality of control actions simultaneously within the cooperative M…
▽ More
In this paper, we discuss the methodology of generalizing the optimal control law from learned component tasks to unlearned composite tasks on Multi-Agent Systems (MASs), by using the linearity composition principle of linearly solvable optimal control (LSOC) problems. The proposed approach achieves both the compositionality and optimality of control actions simultaneously within the cooperative MAS framework in both discrete- and continuous-time in a sample-efficient manner, which reduces the burden of re-computation of the optimal control solutions for the new task on the MASs. We investigate the application of the proposed approach on the MAS with coordination between agents. The experiments show feasible results in investigated scenarios, including both discrete and continuous dynamical systems for task generalization without resampling.
△ Less
Submitted 22 March, 2021; v1 submitted 28 September, 2020;
originally announced September 2020.
-
Contraction $\mathcal{L}_1$-Adaptive Control using Gaussian Processes
Authors:
Aditya Gahlawat,
Arun Lakshmanan,
Lin Song,
Andrew Patterson,
Zhuohuan Wu,
Naira Hovakimyan,
Evangelos Theodorou
Abstract:
We present $\mathcal{CL}_1$-$\mathcal{GP}$, a control framework that enables safe simultaneous learning and control for systems subject to uncertainties. The two main constituents are contraction theory-based $\mathcal{L}_1$ ($\mathcal{CL}_1$) control and Bayesian learning in the form of Gaussian process (GP) regression. The $\mathcal{CL}_1$ controller ensures that control objectives are met while…
▽ More
We present $\mathcal{CL}_1$-$\mathcal{GP}$, a control framework that enables safe simultaneous learning and control for systems subject to uncertainties. The two main constituents are contraction theory-based $\mathcal{L}_1$ ($\mathcal{CL}_1$) control and Bayesian learning in the form of Gaussian process (GP) regression. The $\mathcal{CL}_1$ controller ensures that control objectives are met while providing safety certificates. Furthermore, $\mathcal{CL}_1$-$\mathcal{GP}$ incorporates any available data into a GP model of uncertainties, which improves performance and enables the motion planner to achieve optimality safely. This way, the safe operation of the system is always guaranteed, even during the learning transients. We provide a few illustrative examples for the safe learning and control of planar quadrotor systems in a variety of environments.
△ Less
Submitted 30 November, 2021; v1 submitted 8 September, 2020;
originally announced September 2020.
-
Safe Feedback Motion Planning: A Contraction Theory and $\mathcal{L}_1$-Adaptive Control Based Approach
Authors:
Arun Lakshmanan,
Aditya Gahlawat,
Naira Hovakimyan
Abstract:
Autonomous robots that are capable of operating safely in the presence of imperfect model knowledge or external disturbances are vital in safety-critical applications. In this paper, we present a planner-agnostic framework to design and certify safe tubes around desired trajectories that the robot is always guaranteed to remain inside of. By leveraging recent results in contraction analysis and…
▽ More
Autonomous robots that are capable of operating safely in the presence of imperfect model knowledge or external disturbances are vital in safety-critical applications. In this paper, we present a planner-agnostic framework to design and certify safe tubes around desired trajectories that the robot is always guaranteed to remain inside of. By leveraging recent results in contraction analysis and $\mathcal{L}_1$-adaptive control we synthesize an architecture that induces safe tubes for nonlinear systems with state and time-varying uncertainties. We demonstrate with a few illustrative examples how contraction theory-based $\mathcal{L}_1$-adaptive control can be used in conjunction with traditional motion planning algorithms to obtain provably safe trajectories.
△ Less
Submitted 25 May, 2020; v1 submitted 2 April, 2020;
originally announced April 2020.
-
Learning Probabilistic Intersection Traffic Models for Trajectory Prediction
Authors:
Andrew Patterson,
Aditya Gahlawat,
Naira Hovakimyan
Abstract:
Autonomous agents must be able to safely interact with other vehicles to integrate into urban environments. The safety of these agents is dependent on their ability to predict collisions with other vehicles' future trajectories for replanning and collision avoidance. The information needed to predict collisions can be learned from previously observed vehicle trajectories in a specific environment,…
▽ More
Autonomous agents must be able to safely interact with other vehicles to integrate into urban environments. The safety of these agents is dependent on their ability to predict collisions with other vehicles' future trajectories for replanning and collision avoidance. The information needed to predict collisions can be learned from previously observed vehicle trajectories in a specific environment, generating a traffic model. The learned traffic model can then be incorporated as prior knowledge into any trajectory estimation method being used in this environment. This work presents a Gaussian process based probabilistic traffic model that is used to quantify vehicle behaviors in an intersection. The Gaussian process model provides estimates for the average vehicle trajectory, while also capturing the variance between the different paths a vehicle may take in the intersection. The method is demonstrated on a set of time-series position trajectories. These trajectories are reconstructed by removing object recognition errors and missed frames that may occur due to data source processing. To create the intersection traffic model, the reconstructed trajectories are clustered based on their source and destination lanes. For each cluster, a Gaussian process model is created to capture the average behavior and the variance of the cluster. To show the applicability of the Gaussian model, the test trajectories are classified with only partial observations. Performance is quantified by the number of observations required to correctly classify the vehicle trajectory. Both the intersection traffic modeling computations and the classification procedure are timed. These times are presented as results and demonstrate that the model can be constructed in a reasonable amount of time and the classification procedure can be used for online applications.
△ Less
Submitted 5 February, 2020;
originally announced February 2020.
-
Learning to Communicate: A Machine Learning Framework for Heterogeneous Multi-Agent Robotic Systems
Authors:
Hyung-Jin Yoon,
Huaiyu Chen,
Kehan Long,
Heling Zhang,
Aditya Gahlawat,
Donghwan Lee,
Naira Hovakimyan
Abstract:
We present a machine learning framework for multi-agent systems to learn both the optimal policy for maximizing the rewards and the encoding of the high dimensional visual observation. The encoding is useful for sharing local visual observations with other agents under communication resource constraints. The actor-encoder encodes the raw images and chooses an action based on local observations and…
▽ More
We present a machine learning framework for multi-agent systems to learn both the optimal policy for maximizing the rewards and the encoding of the high dimensional visual observation. The encoding is useful for sharing local visual observations with other agents under communication resource constraints. The actor-encoder encodes the raw images and chooses an action based on local observations and messages sent by the other agents. The machine learning agent generates not only an actuator command to the physical device, but also a communication message to the other agents. We formulate a reinforcement learning problem, which extends the action space to consider the communication action as well. The feasibility of the reinforcement learning framework is demonstrated using a 3D simulation environment with two collaborating agents. The environment provides realistic visual observations to be used and shared between the two agents.
△ Less
Submitted 12 December, 2018;
originally announced December 2018.
