Showing 1–27 of 27 results for author: Herbert, S

Solving Reach- and Stabilize-Avoid Problems Using Discounted Reachability

Authors: Boyang Li, Zheng Gong, Sylvia Herbert

Abstract: In this article, we consider the infinite-horizon reach-avoid (RA) and stabilize-avoid (SA) zero-sum game problems for general nonlinear continuous-time systems, where the goal is to find the set of states that can be controlled to reach or stabilize to a target set, without violating constraints even under the worst-case disturbance. Based on the Hamilton-Jacobi reachability method, we address th… ▽ More In this article, we consider the infinite-horizon reach-avoid (RA) and stabilize-avoid (SA) zero-sum game problems for general nonlinear continuous-time systems, where the goal is to find the set of states that can be controlled to reach or stabilize to a target set, without violating constraints even under the worst-case disturbance. Based on the Hamilton-Jacobi reachability method, we address the RA problem by designing a new Lipschitz continuous RA value function, whose zero sublevel set exactly characterizes the RA set. We establish that the associated Bellman backup operator is contractive and that the RA value function is the unique viscosity solution of a Hamilton-Jacobi variational inequality. Finally, we develop a two-step framework for the SA problem by integrating our RA strategies with a recently proposed Robust Control Lyapunov-Value Function, thereby ensuring both target reachability and long-term stability. We numerically verify our RA and SA frameworks on a 3D Dubins car system to demonstrate the efficacy of the proposed approach. △ Less

Submitted 13 May, 2025; originally announced May 2025.

Comments: 10 pages, 2 figures

Reach-Avoid-Stabilize Using Admissible Control Sets

Authors: Zheng Gong, Boyang Li, Sylvia Herbert

Abstract: Hamilton-Jacobi Reachability (HJR) analysis has been successfully used in many robotics and control tasks, and is especially effective in computing reach-avoid sets and control laws that enable an agent to reach a goal while satisfying state constraints. However, the original HJR formulation provides no guarantees of safety after a) the prescribed time horizon, or b) goal satisfaction. The reach-a… ▽ More Hamilton-Jacobi Reachability (HJR) analysis has been successfully used in many robotics and control tasks, and is especially effective in computing reach-avoid sets and control laws that enable an agent to reach a goal while satisfying state constraints. However, the original HJR formulation provides no guarantees of safety after a) the prescribed time horizon, or b) goal satisfaction. The reach-avoid-stabilize (RAS) problem has therefore gained a lot of focus: find the set of initial states (the RAS set), such that the trajectory can reach the target, and stabilize to some point of interest (POI) while avoiding obstacles. Solving RAS problems using HJR usually requires defining a new value function, whose zero sub-level set is the RAS set. The existing methods do not consider the problem when there are a series of targets to reach and/or obstacles to avoid. We propose a method that uses the idea of admissible control sets; we guarantee that the system will reach each target while avoiding obstacles as prescribed by the given time series. Moreover, we guarantee that the trajectory ultimately stabilizes to the POI. The proposed method provides an under-approximation of the RAS set, guaranteeing safety. Numerical examples are provided to validate the theory. △ Less

Submitted 13 May, 2025; originally announced May 2025.

Comments: 7 pages, 5 figures, submitted to 64th IEEE Conference on Decision and Control

Approximate Hamilton-Jacobi Reachability Analysis for a Class of Two-Timescale Systems, with Application to Biological Models

Authors: Dylan Hirsch, Sylvia Herbert

Abstract: Hamilton-Jacobi reachability (HJR) is an exciting framework used for control of safety-critical systems with nonlinear and possibly uncertain dynamics. However, HJR suffers from the curse of dimensionality, with computation times growing exponentially in the dimension of the system state. Many autonomous and controlled systems involve dynamics that evolve on multiple timescales, and for these syst… ▽ More Hamilton-Jacobi reachability (HJR) is an exciting framework used for control of safety-critical systems with nonlinear and possibly uncertain dynamics. However, HJR suffers from the curse of dimensionality, with computation times growing exponentially in the dimension of the system state. Many autonomous and controlled systems involve dynamics that evolve on multiple timescales, and for these systems, singular perturbation methods can be used for model reduction. However, such methods are more challenging to apply in HJR due to the presence of an underlying differential game. In this work, we leverage prior work on singularly perturbed differential games to identify a class of systems which can be readily reduced, and we relate these results to the quantities of interest in HJR. We demonstrate the utility of our results on two examples involving biological systems, where dynamics fitting the identified class are frequently encountered. △ Less

Submitted 17 March, 2025; v1 submitted 13 March, 2025; originally announced March 2025.

Comments: Second version (note that title changed from previous version)

Back to Base: Towards Hands-Off Learning via Safe Resets with Reach-Avoid Safety Filters

Authors: Azra Begzadić, Nikhil Uday Shinde, Sander Tonkens, Dylan Hirsch, Kaleb Ugalde, Michael C. Yip, Jorge Cortés, Sylvia Herbert

Abstract: Designing controllers that accomplish tasks while guaranteeing safety constraints remains a significant challenge. We often want an agent to perform well in a nominal task, such as environment exploration, while ensuring it can avoid unsafe states and return to a desired target by a specific time. In particular we are motivated by the setting of safe, efficient, hands-off training for reinforcemen… ▽ More Designing controllers that accomplish tasks while guaranteeing safety constraints remains a significant challenge. We often want an agent to perform well in a nominal task, such as environment exploration, while ensuring it can avoid unsafe states and return to a desired target by a specific time. In particular we are motivated by the setting of safe, efficient, hands-off training for reinforcement learning in the real world. By enabling a robot to safely and autonomously reset to a desired region (e.g., charging stations) without human intervention, we can enhance efficiency and facilitate training. Safety filters, such as those based on control barrier functions, decouple safety from nominal control objectives and rigorously guarantee safety. Despite their success, constructing these functions for general nonlinear systems with control constraints and system uncertainties remains an open problem. This paper introduces a safety filter obtained from the value function associated with the reach-avoid problem. The proposed safety filter minimally modifies the nominal controller while avoiding unsafe regions and guiding the system back to the desired target set. By preserving policy performance while allowing safe resetting, we enable efficient hands-off reinforcement learning and advance the feasibility of safe training for real world robots. We demonstrate our approach using a modified version of soft actor-critic to safely train a swing-up task on a modified cartpole stabilization problem. △ Less

Submitted 5 January, 2025; originally announced January 2025.

Comments: The first three authors contributed equally to the work. This work has been submitted to the L4DC 2025 for possible publication

Linear Supervision for Nonlinear, High-Dimensional Neural Control and Differential Games

Authors: William Sharpless, Zeyuan Feng, Somil Bansal, Sylvia Herbert

Abstract: As the dimension of a system increases, traditional methods for control and differential games rapidly become intractable, making the design of safe autonomous agents challenging in complex or team settings. Deep-learning approaches avoid discretization and yield numerous successes in robotics and autonomy, but at a higher dimensional limit, accuracy falls as sampling becomes less efficient. We pr… ▽ More As the dimension of a system increases, traditional methods for control and differential games rapidly become intractable, making the design of safe autonomous agents challenging in complex or team settings. Deep-learning approaches avoid discretization and yield numerous successes in robotics and autonomy, but at a higher dimensional limit, accuracy falls as sampling becomes less efficient. We propose using rapidly generated linear solutions to the partial differential equation (PDE) arising in the problem to accelerate and improve learned value functions for guidance in high-dimensional, nonlinear problems. We define two programs that combine supervision of the linear solution with a standard PDE loss. We demonstrate that these programs offer improvements in speed and accuracy in both a 50-D differential game problem and a 10-D quadrotor control problem. △ Less

Submitted 26 April, 2025; v1 submitted 2 December, 2024; originally announced December 2024.

Safe Barrier-Constrained Control of Uncertain Systems via Event-triggered Learning

Authors: Armin Lederer, Azra Begzadić, Sandra Hirche, Jorge Cortés, Sylvia Herbert

Abstract: While control barrier functions are employed in addressing safety, control synthesis methods based on them generally rely on accurate system dynamics. This is a critical limitation, since the dynamics of complex systems are often not fully known. Supervised machine learning techniques hold great promise for alleviating this weakness by inferring models from data. We propose a novel control barrier… ▽ More While control barrier functions are employed in addressing safety, control synthesis methods based on them generally rely on accurate system dynamics. This is a critical limitation, since the dynamics of complex systems are often not fully known. Supervised machine learning techniques hold great promise for alleviating this weakness by inferring models from data. We propose a novel control barrier function-based framework for safe control through event-triggered learning, which switches between prioritizing control performance and improving model accuracy based on the uncertainty of the learned model. By updating a Gaussian process model with training points gathered online, the approach guarantees the feasibility of control barrier function conditions with high probability, such that safety can be ensured in a data-efficient manner. Furthermore, we establish the absence of Zeno behavior in the triggering scheme, and extend the algorithm to sampled-data realizations by accounting for inter-sampling effects. The effectiveness of the proposed approach and theory is demonstrated in simulations. △ Less

Submitted 31 August, 2024; v1 submitted 28 August, 2024; originally announced August 2024.

Comments: The first two authors contributed equally to the work

Hamilton-Jacobi Reachability in Reinforcement Learning: A Survey

Authors: Milan Ganai, Sicun Gao, Sylvia Herbert

Abstract: Recent literature has proposed approaches that learn control policies with high performance while maintaining safety guarantees. Synthesizing Hamilton-Jacobi (HJ) reachable sets has become an effective tool for verifying safety and supervising the training of reinforcement learning-based control policies for complex, high-dimensional systems. Previously, HJ reachability was restricted to verifying… ▽ More Recent literature has proposed approaches that learn control policies with high performance while maintaining safety guarantees. Synthesizing Hamilton-Jacobi (HJ) reachable sets has become an effective tool for verifying safety and supervising the training of reinforcement learning-based control policies for complex, high-dimensional systems. Previously, HJ reachability was restricted to verifying low-dimensional dynamical systems primarily because the computational complexity of the dynamic programming approach it relied on grows exponentially with the number of system states. In recent years, a litany of proposed methods addresses this limitation by computing the reachability value function simultaneously with learning control policies to scale HJ reachability analysis while still maintaining a reliable estimate of the true reachable set. These HJ reachability approximations are used to improve the safety, and even reward performance, of learned control policies and can solve challenging tasks such as those with dynamic obstacles and/or with lidar-based or vision-based observations. In this survey paper, we review the recent developments in the field of HJ reachability estimation in reinforcement learning that would provide a foundational basis for further research into reliability in high-dimensional systems. △ Less

Submitted 21 August, 2024; v1 submitted 12 July, 2024; originally announced July 2024.

Comments: Accepted in IEEE Open Journal of Control Systems (OJ-CSYS)

Sensor-Based Distributionally Robust Control for Safe Robot Navigation in Dynamic Environments

Authors: Kehan Long, Yinzhuang Yi, Zhirui Dai, Sylvia Herbert, Jorge Cortés, Nikolay Atanasov

Abstract: We introduce a novel method for mobile robot navigation in dynamic, unknown environments, leveraging onboard sensing and distributionally robust optimization to impose probabilistic safety constraints. Our method introduces a distributionally robust control barrier function (DR-CBF) that directly integrates noisy sensor measurements and state estimates to define safety constraints. This approach i… ▽ More We introduce a novel method for mobile robot navigation in dynamic, unknown environments, leveraging onboard sensing and distributionally robust optimization to impose probabilistic safety constraints. Our method introduces a distributionally robust control barrier function (DR-CBF) that directly integrates noisy sensor measurements and state estimates to define safety constraints. This approach is applicable to a wide range of control-affine dynamics, generalizable to robots with complex geometries, and capable of operating at real-time control frequencies. Coupled with a control Lyapunov function (CLF) for path following, the proposed CLF-DR-CBF control synthesis method achieves safe, robust, and efficient navigation in challenging environments. We demonstrate the effectiveness and robustness of our approach for safe autonomous navigation under uncertainty in simulations and real-world experiments with differential-drive robots. △ Less

Submitted 5 May, 2025; v1 submitted 28 May, 2024; originally announced May 2024.

Comments: Project page: https://existentialrobotics.org/DRO_Safe_Navigation

Parameterized Fast and Safe Tracking (FaSTrack) using Deepreach

Authors: Hyun Joe Jeong, Zheng Gong, Somil Bansal, Sylvia Herbert

Abstract: Fast and Safe Tracking (FaSTrack) is a modular framework that provides safety guarantees while planning and executing trajectories in real time via value functions of Hamilton-Jacobi (HJ) reachability. These value functions are computed through dynamic programming, which is notorious for being computationally inefficient. Moreover, the resulting trajectory does not adapt online to the environment,… ▽ More Fast and Safe Tracking (FaSTrack) is a modular framework that provides safety guarantees while planning and executing trajectories in real time via value functions of Hamilton-Jacobi (HJ) reachability. These value functions are computed through dynamic programming, which is notorious for being computationally inefficient. Moreover, the resulting trajectory does not adapt online to the environment, such as sudden disturbances or obstacles. DeepReach is a scalable deep learning method to HJ reachability that allows parameterization of states, which opens up possibilities for online adaptation to various controls and disturbances. In this paper, we propose Parametric FaSTrack, which uses DeepReach to approximate a value function that parameterizes the control bounds of the planning model. The new framework can smoothly trade off between the navigation speed and the tracking error (therefore maneuverability) while guaranteeing obstacle avoidance in a priori unknown environments. We demonstrate our method through two examples and a benchmark comparison with existing methods, showing the safety, efficiency, and faster solution times of the framework. △ Less

Submitted 10 April, 2024; originally announced April 2024.

Comments: 12 pages, 6 figures, 1 table, to be published in L4DC

Safe Returning FaSTrack with Robust Control Lyapunov-Value Functions

Authors: Zheng Gong, Boyang Li, Sylvia Herbert

Abstract: Real-time navigation in a priori unknown environment remains a challenging task, especially when an unexpected (unmodeled) disturbance occurs. In this paper, we propose the framework Safe Returning Fast and Safe Tracking (SR-F) that merges concepts from 1) Robust Control Lyapunov-Value Functions (R-CLVF), and 2) the Fast and Safe Tracking (FaSTrack) framework. The SR-F computes an R-CLVF offline b… ▽ More Real-time navigation in a priori unknown environment remains a challenging task, especially when an unexpected (unmodeled) disturbance occurs. In this paper, we propose the framework Safe Returning Fast and Safe Tracking (SR-F) that merges concepts from 1) Robust Control Lyapunov-Value Functions (R-CLVF), and 2) the Fast and Safe Tracking (FaSTrack) framework. The SR-F computes an R-CLVF offline between a model of the true system and a simplified planning model. Online, a planning algorithm is used to generate a trajectory in the simplified planning space, and the R-CLVF is used to provide a tracking controller that exponentially stabilizes to the planning model. When an unexpected disturbance occurs, the proposed SR-F algorithm provides a means for the true system to recover to the planning model. We take advantage of this mechanism to induce an artificial disturbance by ``jumping'' the planning model in open environments, forcing faster navigation. Therefore, this algorithm can both reject unexpected true disturbances and accelerate navigation speed. We validate our framework using a 10D quadrotor system and show that SR-F is empirically 20\% faster than the original FaSTrack while maintaining safety. △ Less

Submitted 3 April, 2024; originally announced April 2024.

Comments: 6 pages, 4 figures, 1 table, 2 algorithms. Submitted to LCSS on 03/06

Synthesizing Control Lyapunov-Value Functions for High-Dimensional Systems Using System Decomposition and Admissible Control Sets

Authors: Zheng Gong, Hyun Joe Jeong, Sylvia Herbert

Abstract: Control Lyapunov functions (CLFs) play a vital role in modern control applications, but finding them remains a problem. Recently, the control Lyapunov-value function (CLVF) and robust CLVF have been proposed as solutions for nonlinear time-invariant systems with bounded control and disturbance. However, the CLVF suffers from the ''curse of dimensionality,'' which hinders its application to practic… ▽ More Control Lyapunov functions (CLFs) play a vital role in modern control applications, but finding them remains a problem. Recently, the control Lyapunov-value function (CLVF) and robust CLVF have been proposed as solutions for nonlinear time-invariant systems with bounded control and disturbance. However, the CLVF suffers from the ''curse of dimensionality,'' which hinders its application to practical high-dimensional systems. In this paper, we propose a method to decompose systems of a particular coupled nonlinear structure, in order to solve for the CLVF in each low-dimensional subsystem. We then reconstruct the full-dimensional CLVF and provide sufficient conditions for when this reconstruction is exact. Moreover, a point-wise optimal controller can be obtained using a quadratic program. We also show that when the exact reconstruction is impossible, the subsystems' CLVFs and their ``admissible control sets'' can be used to generate a Lipschitz continuous CLF. We provide several numerical examples to validate the theory and show computational efficiency. △ Less

Submitted 2 April, 2024; originally announced April 2024.

Comments: 7 pages, 4 figures, submitted to 63rd Conference on Decision and Control

State-Augmented Linear Games with Antagonistic Error for High-Dimensional, Nonlinear Hamilton-Jacobi Reachability

Authors: Will Sharpless, Yat Tin Chow, Sylvia Herbert

Abstract: Hamilton-Jacobi Reachability (HJR) is a popular method for analyzing the liveness and safety of a dynamical system with bounded control and disturbance. The corresponding HJ value function offers a robust controller and characterizes the reachable sets, but is traditionally solved with Dynamic Programming (DP) and limited to systems of dimension less than six. Recently, the space-parallelizeable,… ▽ More Hamilton-Jacobi Reachability (HJR) is a popular method for analyzing the liveness and safety of a dynamical system with bounded control and disturbance. The corresponding HJ value function offers a robust controller and characterizes the reachable sets, but is traditionally solved with Dynamic Programming (DP) and limited to systems of dimension less than six. Recently, the space-parallelizeable, generalized Hopf formula has been shown to also solve the HJ value with a nearly three-log increase in dimension limit, but is limited to linear systems. To extend this potential, we demonstrate how state-augmented (SA) spaces, which are well-known for their improved linearization accuracy, may be used to solve tighter, conservative approximations of the value function with any linear model in this SA space. Namely, we show that with a representation of the true dynamics in the SA space, a series of inequalities confirms that the value of a SA linear game with antagonistic error is a conservative envelope of the true value function. It follows that if the optimal controller for the HJ SA linear game with error may succeed, it will also succeed in the true system. Unlike previous methods, this result offers the ability to safely approximate reachable sets and their corresponding controllers with the Hopf formula in a non-convex manner. Finally, we demonstrate this in the slow manifold system for clarity, and in the controlled Van der Pol system with different lifting functions. △ Less

Submitted 11 September, 2024; v1 submitted 25 March, 2024; originally announced March 2024.

Conservative Linear Envelopes for Nonlinear, High-Dimensional, Hamilton-Jacobi Reachability

Authors: Will Sharpless, Yat Tin Chow, Sylvia Herbert

Abstract: Hamilton-Jacobi reachability (HJR) provides a value function that encodes the set of states from which a system with bounded control inputs can reach or avoid a target despite any bounded disturbance, and the corresponding robust, optimal control policy. Though powerful, traditional methods for HJR rely on dynamic programming (DP) and suffer from exponential computation growth with respect to stat… ▽ More Hamilton-Jacobi reachability (HJR) provides a value function that encodes the set of states from which a system with bounded control inputs can reach or avoid a target despite any bounded disturbance, and the corresponding robust, optimal control policy. Though powerful, traditional methods for HJR rely on dynamic programming (DP) and suffer from exponential computation growth with respect to state dimension. The recently favored Hopf formula mitigates this ``curse of dimensionality'' by providing an efficient and space-parallelizable approach for solving the reachability problem. However, the Hopf formula can only be applied to linear time-varying systems. To overcome this limitation, we show that the error between a nonlinear system and a linear model can be transformed into an adversarial bounded artificial disturbance. One may then solve the dimension-robust generalized Hopf formula for a linear game with this ``antagonistic error" to perform guaranteed conservative reachability analysis and control synthesis of nonlinear systems; this can be done for problem formulations in which no other HJR method is both computationally feasible and guaranteed. In addition, we offer several technical methods for reducing conservativeness in the analysis. We demonstrate the effectiveness of our results through one illustrative example (the controlled Van der Pol system) that can be compared to standard DP, and one higher-dimensional 15D example (a 5-agent pursuit-evasion game with Dubins cars). △ Less

Submitted 12 April, 2024; v1 submitted 21 March, 2024; originally announced March 2024.

Robust Control Lyapunov-Value Functions for Nonlinear Disturbed Systems

Authors: Zheng Gong, Sylvia Herbert

Abstract: Control Lyapunov Functions (CLFs) have been extensively used in the control community. A well-known drawback is the absence of a systematic way to construct CLFs for general nonlinear systems, and the problem can become more complex with input or state constraints. Our preliminary work on constructing Control Lyapunov Value Functions (CLVFs) using Hamilton-Jacobi (HJ) reachability analysis provide… ▽ More Control Lyapunov Functions (CLFs) have been extensively used in the control community. A well-known drawback is the absence of a systematic way to construct CLFs for general nonlinear systems, and the problem can become more complex with input or state constraints. Our preliminary work on constructing Control Lyapunov Value Functions (CLVFs) using Hamilton-Jacobi (HJ) reachability analysis provides a method for finding a non-smooth CLF. In this paper, we extend our work on CLVFs to systems with bounded disturbance and define the Robust CLVF (R-CLVF). The R-CLVF naturally inherits all properties of the CLVF; i.e., it first identifies the "smallest robust control invariant set (SRCIS)" and stabilizes the system to it with a user-specified exponential rate. The region from which the exponential rate can be met is called the "region of exponential stabilizability (ROES)." We provide clearer definitions of the SRCIS and more rigorous proofs of several important theorems. Since the computation of the R-CLVF suffers from the "curse of dimensionality," we also provide two techniques (warmstart and system decomposition) that solve it, along with necessary proofs. Three numerical examples are provided, validating our definition of SRCIS, illustrating the trade-off between a faster decay rate and a smaller ROES, and demonstrating the efficiency of computation using warmstart and decomposition. △ Less

Submitted 5 March, 2024; originally announced March 2024.

Comments: 13 pages, 5 figures

A Forward Reachability Perspective on Robust Control Invariance and Discount Factors in Reachability Analysis

Authors: Jason J. Choi, Donggun Lee, Boyang Li, Jonathan P. How, Koushil Sreenath, Sylvia L. Herbert, Claire J. Tomlin

Abstract: Control invariant sets are crucial for various methods that aim to design safe control policies for systems whose state constraints must be satisfied over an indefinite time horizon. In this article, we explore the connections among reachability, control invariance, and Control Barrier Functions (CBFs). Unlike prior formulations based on backward reachability concepts, by examining a forward reach… ▽ More Control invariant sets are crucial for various methods that aim to design safe control policies for systems whose state constraints must be satisfied over an indefinite time horizon. In this article, we explore the connections among reachability, control invariance, and Control Barrier Functions (CBFs). Unlike prior formulations based on backward reachability concepts, by examining a forward reachability problem, we are able to establish a strong link between these three concepts. First, our findings show that the inevitable Forward Reachable Tube (FRT), which is the set of states such that every trajectory reaching the FRT must have passed through a given initial set of states, is precisely this initial set of states itself if it is a robust control invariant set with a differentiable boundary. We highlight that this statement may not hold if the boundary is not differentiable. Next, we formulate a differential game between the control and disturbance, where the inevitable FRT is characterized by the zero-superlevel set of the value function. By incorporating a discount factor in the cost function of the game, the barrier constraint of the CBF naturally arises as the constraint that is imposed on the optimal control policy. Combining these results, the value function of our FRT formulation serves as a CBF-like function, and conversely, any valid CBF is also a forward reachability value function inside the control invariant set, thereby revealing the inverse optimality of the CBF. This strong link we establish between the reachability problem and the barrier constraint, while guaranteeing the continuity of the value function, is not achievable by previous backward reachability-based formulations. As such, our work fills a crucial gap in the existing literature that is vital for constructing valid CBFs to ensure safety. △ Less

Submitted 28 July, 2024; v1 submitted 26 October, 2023; originally announced October 2023.

Comments: The first two authors contributed equally to this work

A Control Approach for Nonlinear Stochastic State Uncertain Systems with Probabilistic Safety Guarantees

Authors: Mohammad S. Ramadan, Mohammad Alsuwaidan, Ahmed Atallah, Sylvia Herbert

Abstract: This paper presents an algorithm to apply nonlinear control design approaches in the case of stochastic systems with partial state observation. Deterministic nonlinear control approaches are formulated under the assumption of full state access and, often, relative degree one. We propose a control design approach that first generates a control policy for nonlinear deterministic models with full sta… ▽ More This paper presents an algorithm to apply nonlinear control design approaches in the case of stochastic systems with partial state observation. Deterministic nonlinear control approaches are formulated under the assumption of full state access and, often, relative degree one. We propose a control design approach that first generates a control policy for nonlinear deterministic models with full state observation. The resulting control policy is then used to build an importance-like probability distribution over the space of control sequences which are to be evaluated for the true stochastic and state-uncertain dynamics. This distribution serves in the sampling step within a random search control optimization procedure, to focus the exploration effort on certain regions of the control space. The sampled control sequences are assigned costs determined by a prescribed finite-horizon performance and safety measure, which is based on the stochastic dynamics. This sampling algorithm is parallelizable and shown to have computational complexity indifferent to the state dimension, and to be able to guarantee safety over the prescribed prediction horizon. A numerical simulation is provided to test the applicability and effectiveness of the presented approach and compare it to a certainty equivalence controller. △ Less

Submitted 15 September, 2023; originally announced September 2023.

Patching Approximately Safe Value Functions Leveraging Local Hamilton-Jacobi Reachability Analysis

Authors: Sander Tonkens, Alex Toofanian, Zhizhen Qin, Sicun Gao, Sylvia Herbert

Abstract: Safe value functions, such as control barrier functions, characterize a safe set and synthesize a safety filter, overriding unsafe actions, for a dynamic system. While function approximators like neural networks can synthesize approximately safe value functions, they typically lack formal guarantees. In this paper, we propose a local dynamic programming-based approach to "patch" approximately safe… ▽ More Safe value functions, such as control barrier functions, characterize a safe set and synthesize a safety filter, overriding unsafe actions, for a dynamic system. While function approximators like neural networks can synthesize approximately safe value functions, they typically lack formal guarantees. In this paper, we propose a local dynamic programming-based approach to "patch" approximately safe value functions to obtain a safe value function. This algorithm, HJ-Patch, produces a novel value function that provides formal safety guarantees, yet retains the global structure of the initial value function. HJ-Patch modifies an approximately safe value function at states that are both (i) near the safety boundary and (ii) may violate safety. We iteratively update both this set of "active" states and the value function until convergence. This approach bridges the gap between value function approximation methods and formal safety through Hamilton-Jacobi (HJ) reachability, offering a framework for integrating various safety methods. We provide simulation results on analytic and learned examples, demonstrating HJ-Patch reduces the computational complexity by 2 orders of magnitude with respect to standard HJ reachability. Additionally, we demonstrate the perils of using approximately safe value functions directly and showcase improved safety using HJ-Patch. △ Less

Submitted 6 September, 2024; v1 submitted 19 April, 2023; originally announced April 2023.

Comments: 8 pages, IEEE Conference on Decision and Control (CDC), 2024 (In Press)

Koopman-Hopf Hamilton-Jacobi Reachability and Control

Authors: Will Sharpless, Nikhil Shinde, Matthew Kim, Yat Tin Chow, Sylvia Herbert

Abstract: The Hopf formula for Hamilton-Jacobi reachability (HJR) analysis has been proposed to solve high-dimensional differential games, producing the set of initial states and corresponding controller required to reach (or avoid) a target despite bounded disturbances. As a space-parallelizable method, the Hopf formula avoids the curse of dimensionality that afflicts standard dynamic-programming HJR, but… ▽ More The Hopf formula for Hamilton-Jacobi reachability (HJR) analysis has been proposed to solve high-dimensional differential games, producing the set of initial states and corresponding controller required to reach (or avoid) a target despite bounded disturbances. As a space-parallelizable method, the Hopf formula avoids the curse of dimensionality that afflicts standard dynamic-programming HJR, but is restricted to linear time-varying systems. To compute reachable sets for high-dimensional nonlinear systems, we pair the Hopf solution with Koopman theory for global linearization. By first lifting a nonlinear system to a linear space and then solving the Hopf formula, approximate reachable sets can be efficiently computed that are much more accurate than local linearizations. Furthermore, we construct a Koopman-Hopf disturbance-rejecting controller, and test its ability to drive a 10-dimensional nonlinear glycolysis model. We find that it significantly out-competes expectation-minimizing and game-theoretic model predictive controllers with the same Koopman linearization in the presence of bounded stochastic disturbance. In summary, we demonstrate a dimension-robust method to approximately solve HJR, allowing novel application to analyze and control high-dimensional, nonlinear systems with disturbance. An open-source toolbox in Julia is introduced for both Hopf and Koopman-Hopf reachability and control. △ Less

Submitted 24 August, 2023; v1 submitted 21 March, 2023; originally announced March 2023.

Refining Control Barrier Functions through Hamilton-Jacobi Reachability

Authors: Sander Tonkens, Sylvia Herbert

Abstract: Safety filters based on Control Barrier Functions (CBFs) have emerged as a practical tool for the safety-critical control of autonomous systems. These approaches encode safety through a value function and enforce safety by imposing a constraint on the time derivative of this value function. However, synthesizing a valid CBF that is not overly conservative in the presence of input constraints is a… ▽ More Safety filters based on Control Barrier Functions (CBFs) have emerged as a practical tool for the safety-critical control of autonomous systems. These approaches encode safety through a value function and enforce safety by imposing a constraint on the time derivative of this value function. However, synthesizing a valid CBF that is not overly conservative in the presence of input constraints is a notorious challenge. In this work, we propose refining a candidate CBF using formal verification methods to obtain a valid CBF. In particular, we update an expert-synthesized or backup CBF using dynamic programming (DP) based reachability analysis. Our framework, refineCBF, guarantees that with every DP iteration the obtained CBF is provably at least as safe as the prior iteration and converges to a valid CBF. Therefore, refineCBF can be used in-the-loop for robotic systems. We demonstrate the practicality of our method to enhance safety and/or reduce conservativeness on a range of nonlinear control-affine systems using various CBF synthesis techniques in simulation. △ Less

Submitted 19 August, 2022; v1 submitted 26 April, 2022; originally announced April 2022.

Comments: To appear at IEEE/RSJ International Conference on Intelligent Robots & Systems (IROS) 2022

A Second-Order Reachable Sets Computation Scheme via a Cauchy-Type Variational Hamilton-Jacobi-Isaacs Equation

Authors: Lekan Molu, Ian Abraham, Sylvia Herbert

Abstract: Motivated by the scalability limitations of Eulerian methods for variational Hamilton-Jacobi-Isaacs (HJI) formulations that provide a least restrictive controller in problems that involve state or input constraints under a worst-possible disturbance, we introduce a second-order, successive sweep algorithm for computing the zero sublevel sets of a popular reachability value functional. Under suffic… ▽ More Motivated by the scalability limitations of Eulerian methods for variational Hamilton-Jacobi-Isaacs (HJI) formulations that provide a least restrictive controller in problems that involve state or input constraints under a worst-possible disturbance, we introduce a second-order, successive sweep algorithm for computing the zero sublevel sets of a popular reachability value functional. Under sufficient HJI partial differential equation regularity and continuity assumption throughout the state space, we show that with state feedback control under the worst-possible disturbance, we can compute the state set that are reachable within a prescribed verification time bound. △ Less

Submitted 22 June, 2022; v1 submitted 8 March, 2022; originally announced March 2022.

Comments: 6 pages

Safe Autonomous Racing via Approximate Reachability on Ego-vision

Authors: Bingqing Chen, Jonathan Francis, Jean Oh, Eric Nyberg, Sylvia L. Herbert

Abstract: Racing demands each vehicle to drive at its physical limits, when any safety infraction could lead to catastrophic failure. In this work, we study the problem of safe reinforcement learning (RL) for autonomous racing, using the vehicle's ego-camera view and speed as input. Given the nature of the task, autonomous agents need to be able to 1) identify and avoid unsafe scenarios under the complex ve… ▽ More Racing demands each vehicle to drive at its physical limits, when any safety infraction could lead to catastrophic failure. In this work, we study the problem of safe reinforcement learning (RL) for autonomous racing, using the vehicle's ego-camera view and speed as input. Given the nature of the task, autonomous agents need to be able to 1) identify and avoid unsafe scenarios under the complex vehicle dynamics, and 2) make sub-second decision in a fast-changing environment. To satisfy these criteria, we propose to incorporate Hamilton-Jacobi (HJ) reachability theory, a safety verification method for general non-linear systems, into the constrained Markov decision process (CMDP) framework. HJ reachability not only provides a control-theoretic approach to learn about safety, but also enables low-latency safety verification. Though HJ reachability is traditionally not scalable to high-dimensional systems, we demonstrate that with neural approximation, the HJ safety value can be learned directly on vision context -- the highest-dimensional problem studied via the method, to-date. We evaluate our method on several benchmark tasks, including Safety Gym and Learn-to-Race (L2R), a recently-released high-fidelity autonomous racing environment. Our approach has significantly fewer constraint violations in comparison to other constrained RL baselines in Safety Gym, and achieves the new state-of-the-art results on the L2R benchmark task. We provide additional visualization of agent behavior at the following anonymized paper website: https://sites.google.com/view/safeautonomousracing/home △ Less

Submitted 30 November, 2021; v1 submitted 14 October, 2021; originally announced October 2021.

Comments: 17 pages, 15 figures, 3 tables

Robust Control Barrier-Value Functions for Safety-Critical Control

Authors: Jason J. Choi, Donggun Lee, Koushil Sreenath, Claire J. Tomlin, Sylvia L. Herbert

Abstract: This paper works towards unifying two popular approaches in the safety control community: Hamilton-Jacobi (HJ) reachability and Control Barrier Functions (CBFs). HJ Reachability has methods for direct construction of value functions that provide safety guarantees and safe controllers, however the online implementation can be overly conservative and/or rely on chattering bang-bang control. The CBF… ▽ More This paper works towards unifying two popular approaches in the safety control community: Hamilton-Jacobi (HJ) reachability and Control Barrier Functions (CBFs). HJ Reachability has methods for direct construction of value functions that provide safety guarantees and safe controllers, however the online implementation can be overly conservative and/or rely on chattering bang-bang control. The CBF community has methods for safe-guarding controllers in the form of point-wise optimization using quadratic programs (CBF-QP), where the CBF-based safety certificate is used as a constraint. However, finding a valid CBF for a general dynamical system is challenging. This paper unifies these two methods by introducing a new reachability formulation inspired by the structure of CBFs to construct a Control Barrier-Value Function (CBVF). We verify that CBVF is a viscosity solution to a novel Hamilton-Jacobi-Isaacs Variational Inequality and preserves the same safety guarantee as the original reachability formulation. Finally, inspired by the CBF-QP, we propose a QP-based online control synthesis for systems affine in control and disturbance, whose solution is always the CBVF's optimal control signal robust to bounded disturbance. We demonstrate the benefit of using the CBVFs for double-integrator and Dubins car systems by comparing it to previous methods. △ Less

Submitted 25 October, 2021; v1 submitted 6 April, 2021; originally announced April 2021.

Comments: IEEE CDC 2021

Scalable Learning of Safety Guarantees for Autonomous Systems using Hamilton-Jacobi Reachability

Authors: Sylvia Herbert, Jason J. Choi, Suvansh Sanjeev, Marsalis Gibson, Koushil Sreenath, Claire J. Tomlin

Abstract: Autonomous systems like aircraft and assistive robots often operate in scenarios where guaranteeing safety is critical. Methods like Hamilton-Jacobi reachability can provide guaranteed safe sets and controllers for such systems. However, often these same scenarios have unknown or uncertain environments, system dynamics, or predictions of other agents. As the system is operating, it may learn new k… ▽ More Autonomous systems like aircraft and assistive robots often operate in scenarios where guaranteeing safety is critical. Methods like Hamilton-Jacobi reachability can provide guaranteed safe sets and controllers for such systems. However, often these same scenarios have unknown or uncertain environments, system dynamics, or predictions of other agents. As the system is operating, it may learn new knowledge about these uncertainties and should therefore update its safety analysis accordingly. However, work to learn and update safety analysis is limited to small systems of about two dimensions due to the computational complexity of the analysis. In this paper we synthesize several techniques to speed up computation: decomposition, warm-starting, and adaptive grids. Using this new framework we can update safe sets by one or more orders of magnitude faster than prior work, making this technique practical for many realistic systems. We demonstrate our results on simulated 2D and 10D near-hover quadcopters operating in a windy environment. △ Less

Submitted 2 April, 2021; v1 submitted 14 January, 2021; originally announced January 2021.

Comments: The first two authors are co-first authors. ICRA 2021

Reachability-Based Safety Guarantees using Efficient Initializations

Authors: Sylvia L. Herbert, Shromona Ghosh, Somil Bansal, Claire J. Tomlin

Abstract: Hamilton-Jacobi-Isaacs (HJI) reachability analysis is a powerful tool for analyzing the safety of autonomous systems. This analysis is computationally intensive and typically performed offline. Online, however, the autonomous system may experience changes in system dynamics, external disturbances, and/or the surrounding environment, requiring updated safety guarantees. Rather than restarting the s… ▽ More Hamilton-Jacobi-Isaacs (HJI) reachability analysis is a powerful tool for analyzing the safety of autonomous systems. This analysis is computationally intensive and typically performed offline. Online, however, the autonomous system may experience changes in system dynamics, external disturbances, and/or the surrounding environment, requiring updated safety guarantees. Rather than restarting the safety analysis, we propose a method of "warm-start" reachability, which uses a user-defined initialization (typically the previously computed solution). By starting with an HJI function that is closer to the solution than the standard initialization, convergence may take fewer iterations. In this paper we prove that warm-starting will result in guaranteed conservative solutions by over-approximating the states that must be avoided to maintain safety. We additionally prove that for many common problem formulations, warm-starting will result in exact solutions.We demonstrate our method on several illustrative examples with a double integrator, and also on a more practical example with a 10D quadcopter model that experiences changes in mass and disturbances and must update its safety guarantees accordingly. We compare our approach to standard reachability and a recently proposed "discounted" reachability method, and find for our examples that warm-starting is 1.6 times faster than standard and 6.2 times faster than (untuned) discounted reachability. △ Less

Submitted 18 March, 2019; originally announced March 2019.

Comments: Submitted to the 2019 IEEE Conference on Decision and Control, 8 pages, 3 figures

Robust Tracking with Model Mismatch for Fast and Safe Planning: an SOS Optimization Approach

Authors: Sumeet Singh, Mo Chen, Sylvia L. Herbert, Claire J. Tomlin, Marco Pavone

Abstract: In the pursuit of real-time motion planning, a commonly adopted practice is to compute a trajectory by running a planning algorithm on a simplified, low-dimensional dynamical model, and then employ a feedback tracking controller that tracks such a trajectory by accounting for the full, high-dimensional system dynamics. While this strategy of planning with model mismatch generally yields fast compu… ▽ More In the pursuit of real-time motion planning, a commonly adopted practice is to compute a trajectory by running a planning algorithm on a simplified, low-dimensional dynamical model, and then employ a feedback tracking controller that tracks such a trajectory by accounting for the full, high-dimensional system dynamics. While this strategy of planning with model mismatch generally yields fast computation times, there are no guarantees of dynamic feasibility, which hampers application to safety-critical systems. Building upon recent work that addressed this problem through the lens of Hamilton-Jacobi (HJ) reachability, we devise an algorithmic framework whereby one computes, offline, for a pair of "planner" (i.e., low-dimensional) and "tracking" (i.e., high-dimensional) models, a feedback tracking controller and associated tracking bound. This bound is then used as a safety margin when generating motion plans via the low-dimensional model. Specifically, we harness the computational tool of sum-of-squares (SOS) programming to design a bilinear optimization algorithm for the computation of the feedback tracking controller and associated tracking bound. The algorithm is demonstrated via numerical experiments, with an emphasis on investigating the trade-off between the increased computational scalability afforded by SOS and its intrinsic conservativeness. Collectively, our results enable scaling the appealing strategy of planning with model mismatch to systems that are beyond the reach of HJ analysis, while maintaining safety guarantees. △ Less

Submitted 28 July, 2019; v1 submitted 1 August, 2018; originally announced August 2018.

Comments: Presented at WAFR 2018; final version v2 -- fixed typos

Planning, Fast and Slow: A Framework for Adaptive Real-Time Safe Trajectory Planning

Authors: David Fridovich-Keil, Sylvia L. Herbert, Jaime F. Fisac, Sampada Deglurkar, Claire J. Tomlin

Abstract: Motion planning is an extremely well-studied problem in the robotics community, yet existing work largely falls into one of two categories: computationally efficient but with few if any safety guarantees, or able to give stronger guarantees but at high computational cost. This work builds on a recent development called FaSTrack in which a slow offline computation provides a modular safety guarante… ▽ More Motion planning is an extremely well-studied problem in the robotics community, yet existing work largely falls into one of two categories: computationally efficient but with few if any safety guarantees, or able to give stronger guarantees but at high computational cost. This work builds on a recent development called FaSTrack in which a slow offline computation provides a modular safety guarantee for a faster online planner. We introduce the notion of "meta-planning" in which a refined offline computation enables safe switching between different online planners. This provides autonomous systems with the ability to adapt motion plans to a priori unknown environments in real-time as sensor measurements detect new obstacles, and the flexibility to maneuver differently in the presence of obstacles than they would in free space, all while maintaining a strict safety guarantee. We demonstrate the meta-planning algorithm both in simulation and in hardware using a small Crazyflie 2.0 quadrotor. △ Less

Submitted 6 March, 2018; v1 submitted 12 October, 2017; originally announced October 2017.

Comments: ICRA, International Conference on Robotics and Automation, ICRA 2018, 8 pages, 9 figures

Hamilton-Jacobi Reachability: A Brief Overview and Recent Advances

Authors: Somil Bansal, Mo Chen, Sylvia Herbert, Claire J. Tomlin

Abstract: Hamilton-Jacobi (HJ) reachability analysis is an important formal verification method for guaranteeing performance and safety properties of dynamical systems; it has been applied to many small-scale systems in the past decade. Its advantages include compatibility with general nonlinear system dynamics, formal treatment of bounded disturbances, and the availability of well-developed numerical tools… ▽ More Hamilton-Jacobi (HJ) reachability analysis is an important formal verification method for guaranteeing performance and safety properties of dynamical systems; it has been applied to many small-scale systems in the past decade. Its advantages include compatibility with general nonlinear system dynamics, formal treatment of bounded disturbances, and the availability of well-developed numerical tools. The main challenge is addressing its exponential computational complexity with respect to the number of state variables. In this tutorial, we present an overview of basic HJ reachability theory and provide instructions for using the most recent numerical tools, including an efficient GPU-parallelized implementation of a Level Set Toolbox for computing reachable sets. In addition, we review some of the current work in high-dimensional HJ reachability to show how the dimensionality challenge can be alleviated via various general theoretical and application-specific insights. △ Less

Submitted 21 September, 2017; originally announced September 2017.

Comments: This paper summarizes the recent major research efforts in reachability. The paper will be presented in the proceedings of CDC 2017