-
Safe Reinforcement Learning for an Energy-Efficient Driver Assistance System
Authors:
Habtamu Hailemichael,
Beshah Ayalew,
Lindsey Kerbel,
Andrej Ivanco,
Keith Loiselle
Abstract:
Reinforcement learning (RL)-based driver assistance systems seek to improve fuel consumption via continual improvement of powertrain control actions considering experiential data from the field. However, the need to explore diverse experiences in order to learn optimal policies often limits the application of RL techniques in safety-critical systems like vehicle control. In this paper, an exponent…
▽ More
Reinforcement learning (RL)-based driver assistance systems seek to improve fuel consumption via continual improvement of powertrain control actions considering experiential data from the field. However, the need to explore diverse experiences in order to learn optimal policies often limits the application of RL techniques in safety-critical systems like vehicle control. In this paper, an exponential control barrier function (ECBF) is derived and utilized to filter unsafe actions proposed by an RL-based driver assistance system. The RL agent freely explores and optimizes the performance objectives while unsafe actions are projected to the closest actions in the safe domain. The reward is structured so that driver's acceleration requests are met in a manner that boosts fuel economy and doesn't compromise comfort. The optimal gear and traction torque control actions that maximize the cumulative reward are computed via the Maximum a Posteriori Policy Optimization (MPO) algorithm configured for a hybrid action space. The proposed safe-RL scheme is trained and evaluated in car following scenarios where it is shown that it effectively avoids collision both during training and evaluation while delivering on the expected fuel economy improvements for the driver assistance system.
△ Less
Submitted 2 January, 2023;
originally announced January 2023.
-
Safety Filtering for Reinforcement Learning-based Adaptive Cruise Control
Authors:
Habtamu Hailemichael,
Beshah Ayalew,
Lindsey Kerbel,
Andrej Ivanco,
Keith Loiselle
Abstract:
Reinforcement learning (RL)-based adaptive cruise control systems (ACC) that learn and adapt to road, traffic and vehicle conditions are attractive for enhancing vehicle energy efficiency and traffic flow. However, the application of RL in safety critical systems such as ACC requires strong safety guarantees which are difficult to achieve with learning agents that have a fundamental need to explor…
▽ More
Reinforcement learning (RL)-based adaptive cruise control systems (ACC) that learn and adapt to road, traffic and vehicle conditions are attractive for enhancing vehicle energy efficiency and traffic flow. However, the application of RL in safety critical systems such as ACC requires strong safety guarantees which are difficult to achieve with learning agents that have a fundamental need to explore. In this paper, we derive control barrier functions as safety filters that allow an RL-based ACC controller to explore freely within a collision safe set. Specifically, we derive control barrier functions for high relative degree nonlinear systems to take into account inertia effects relevant to commercial vehicles. We also outline an algorithm for accommodating actuation saturation with these barrier functions. While any RL algorithm can be used as the performance ACC controller together with these filters, we implement the Maximum A Posteriori Policy Optimization (MPO) algorithm with a hybrid action space that learns fuel optimal gear selection and torque control policies. The safety filtering RL approach is contrasted with a reward shaping RL approach that only learns to avoid collisions after sufficient training. Evaluations on different drive cycles demonstrate significant improvements in fuel economy with the proposed approach compared to baseline ACC algorithms.
△ Less
Submitted 2 January, 2023;
originally announced January 2023.
-
Residual Policy Learning for Powertrain Control
Authors:
Lindsey Kerbel,
Beshah Ayalew,
Andrej Ivanco,
Keith Loiselle
Abstract:
Eco-driving strategies have been shown to provide significant reductions in fuel consumption. This paper outlines an active driver assistance approach that uses a residual policy learning (RPL) agent trained to provide residual actions to default power train controllers while balancing fuel consumption against other driver-accommodation objectives. Using previous experiences, our RPL agent learns…
▽ More
Eco-driving strategies have been shown to provide significant reductions in fuel consumption. This paper outlines an active driver assistance approach that uses a residual policy learning (RPL) agent trained to provide residual actions to default power train controllers while balancing fuel consumption against other driver-accommodation objectives. Using previous experiences, our RPL agent learns improved traction torque and gear shifting residual policies to adapt the operation of the powertrain to variations and uncertainties in the environment. For comparison, we consider a traditional reinforcement learning (RL) agent trained from scratch. Both agents employ the off-policy Maximum A Posteriori Policy Optimization algorithm with an actor-critic architecture. By implementing on a simulated commercial vehicle in various car-following scenarios, we find that the RPL agent quickly learns significantly improved policies compared to a baseline source policy but in some measures not as good as those eventually possible with the RL agent trained from scratch.
△ Less
Submitted 14 December, 2022;
originally announced December 2022.
-
Driver Assistance Eco-driving and Transmission Control with Deep Reinforcement Learning
Authors:
Lindsey Kerbel,
Beshah Ayalew,
Andrej Ivanco,
Keith Loiselle
Abstract:
With the growing need to reduce energy consumption and greenhouse gas emissions, Eco-driving strategies provide a significant opportunity for additional fuel savings on top of other technological solutions being pursued in the transportation sector. In this paper, a model-free deep reinforcement learning (RL) control agent is proposed for active Eco-driving assistance that trades-off fuel consumpt…
▽ More
With the growing need to reduce energy consumption and greenhouse gas emissions, Eco-driving strategies provide a significant opportunity for additional fuel savings on top of other technological solutions being pursued in the transportation sector. In this paper, a model-free deep reinforcement learning (RL) control agent is proposed for active Eco-driving assistance that trades-off fuel consumption against other driver-accommodation objectives, and learns optimal traction torque and transmission shifting policies from experience. The training scheme for the proposed RL agent uses an off-policy actor-critic architecture that iteratively does policy evaluation with a multi-step return and policy improvement with the maximum posteriori policy optimization algorithm for hybrid action spaces. The proposed Eco-driving RL agent is implemented on a commercial vehicle in car following traffic. It shows superior performance in minimizing fuel consumption compared to a baseline controller that has full knowledge of fuel-efficiency tables.
△ Less
Submitted 14 December, 2022;
originally announced December 2022.
