-
LiMTR: Time Series Motion Prediction for Diverse Road Users through Multimodal Feature Integration
Authors:
Camiel Oerlemans,
Bram Grooten,
Michiel Braat,
Alaa Alassi,
Emilia Silvas,
Decebal Constantin Mocanu
Abstract:
Predicting the behavior of road users accurately is crucial to enable the safe operation of autonomous vehicles in urban or densely populated areas. Therefore, there has been a growing interest in time series motion prediction research, leading to significant advancements in state-of-the-art techniques in recent years. However, the potential of using LiDAR data to capture more detailed local featu…
▽ More
Predicting the behavior of road users accurately is crucial to enable the safe operation of autonomous vehicles in urban or densely populated areas. Therefore, there has been a growing interest in time series motion prediction research, leading to significant advancements in state-of-the-art techniques in recent years. However, the potential of using LiDAR data to capture more detailed local features, such as a person's gaze or posture, remains largely unexplored. To address this, we develop a novel multimodal approach for motion prediction based on the PointNet foundation model architecture, incorporating local LiDAR features. Evaluation on the Waymo Open Dataset shows a performance improvement of 6.20% and 1.58% in minADE and mAP respectively, when integrated and compared with the previous state-of-the-art MTR. We open-source the code of our LiMTR model.
△ Less
Submitted 21 October, 2024;
originally announced October 2024.
-
SDS++: Online Situation-Aware Drivable Space Estimation for Automated Driving
Authors:
Manuel Muñoz Sánchez,
Gijs Trots,
Robin Smit,
Pedro Vieira Oliveira,
Emilia Silvas,
Jos Elfring,
René van de Molengraft
Abstract:
Autonomous Vehicles (AVs) need an accurate and up-to-date representation of the environment for safe navigation. Traditional methods, which often rely on detailed environmental representations constructed offline, struggle in dynamically changing environments or when dealing with outdated maps. Consequently, there is a pressing need for real-time solutions that can integrate diverse data sources a…
▽ More
Autonomous Vehicles (AVs) need an accurate and up-to-date representation of the environment for safe navigation. Traditional methods, which often rely on detailed environmental representations constructed offline, struggle in dynamically changing environments or when dealing with outdated maps. Consequently, there is a pressing need for real-time solutions that can integrate diverse data sources and adapt to the current situation. An existing framework that addresses these challenges is SDS (situation-aware drivable space). However, SDS faces several limitations, including its use of a non-standard output representation, its choice of encoding objects as points, restricting representation of more complex geometries like road lanes, and the fact that its methodology has been validated only with simulated or heavily post-processed data. This work builds upon SDS and introduces SDS++, designed to overcome SDS's shortcomings while preserving its benefits. SDS++ has been rigorously validated not only in simulations but also with unrefined vehicle data, and it is integrated with a model predictive control (MPC)-based planner to verify its advantages for the planning task. The results demonstrate that SDS++ significantly enhances trajectory planning capabilities, providing increased robustness against localization noise, and enabling the planning of trajectories that adapt to the current driving context.
△ Less
Submitted 3 June, 2024;
originally announced June 2024.
-
Characterization and Mitigation of Insufficiencies in Automated Driving Systems
Authors:
Yuting Fu,
Jochen Seemann,
Caspar Hanselaar,
Tim Beurskens,
Andrei Terechko,
Emilia Silvas,
Maurice Heemels
Abstract:
Automated Driving (AD) systems have the potential to increase safety, comfort and energy efficiency. Recently, major automotive companies have started testing and validating AD systems (ADS) on public roads. Nevertheless, the commercial deployment and wide adoption of ADS have been moderate, partially due to system functional insufficiencies (FI) that undermine passenger safety and lead to hazardo…
▽ More
Automated Driving (AD) systems have the potential to increase safety, comfort and energy efficiency. Recently, major automotive companies have started testing and validating AD systems (ADS) on public roads. Nevertheless, the commercial deployment and wide adoption of ADS have been moderate, partially due to system functional insufficiencies (FI) that undermine passenger safety and lead to hazardous situations on the road. FIs are defined in ISO 21448 Safety Of The Intended Functionality (SOTIF). FIs are insufficiencies in sensors, actuators and algorithm implementations, including neural networks and probabilistic calculations. Examples of FIs in ADS include inaccurate ego-vehicle localization on the road, incorrect prediction of a cyclist maneuver, unreliable detection of a pedestrian, etc.
The main goal of our study is to formulate a generic architectural design pattern, which is compatible with existing methods and ADS, to improve FI mitigation and enable faster commercial deployment of ADS. First, we studied the 2021 autonomous vehicles disengagement reports published by the California Department of Motor Vehicles (DMV). The data clearly show that disengagements are five times more often caused by FIs rather than by system faults. We then made a comprehensive list of insufficiencies and their characteristics by analyzing over 10 hours of publicly available road test videos. In particular, we identified insufficiency types in four major categories: world model, motion plan, traffic rule, and operational design domain. The insufficiency characterization helps making the SOTIF analyses of triggering conditions more systematic and comprehensive.
Based on our FI characterization, simulation experiments and literature survey, we define a novel generic architectural design pattern Daruma to dynamically select the channel that is least likely to have a FI at the moment.
△ Less
Submitted 15 April, 2024;
originally announced April 2024.
-
Prediction Horizon Requirements for Automated Driving: Optimizing Safety, Comfort, and Efficiency
Authors:
Manuel Muñoz Sánchez,
Chris van der Ploeg,
Robin Smit,
Jos Elfring,
Emilia Silvas,
René van de Molengraft
Abstract:
Predicting the movement of other road users is beneficial for improving automated vehicle (AV) performance. However, the relationship between the time horizon associated with these predictions and AV performance remains unclear. Despite the existence of numerous trajectory prediction algorithms, no studies have been conducted on how varying prediction lengths affect AV safety and other vehicle per…
▽ More
Predicting the movement of other road users is beneficial for improving automated vehicle (AV) performance. However, the relationship between the time horizon associated with these predictions and AV performance remains unclear. Despite the existence of numerous trajectory prediction algorithms, no studies have been conducted on how varying prediction lengths affect AV safety and other vehicle performance metrics, resulting in undefined horizon requirements for prediction methods. Our study addresses this gap by examining the effects of different prediction horizons on AV performance, focusing on safety, comfort, and efficiency. Through multiple experiments using a state-of-the-art, risk-based predictive trajectory planner, we simulated predictions with horizons up to 20 seconds. Based on our simulations, we propose a framework for specifying the minimum required and optimal prediction horizons based on specific AV performance criteria and application needs. Our results indicate that a horizon of 1.6 seconds is required to prevent collisions with crossing pedestrians, horizons of 7-8 seconds yield the best efficiency, and horizons up to 15 seconds improve passenger comfort. We conclude that prediction horizon requirements are application-dependent, and recommend aiming for a prediction horizon of 11.8 seconds as a general guideline for applications involving crossing pedestrians.
△ Less
Submitted 10 April, 2024; v1 submitted 6 February, 2024;
originally announced February 2024.
-
The Safety Shell: an Architecture to Handle Functional Insufficiencies in Automated Driving
Authors:
C. A. J. Hanselaar,
E. Silvas,
A. Terechko,
W. P. M. H. Heemels
Abstract:
To enable highly automated vehicles where the driver is no longer a safety backup, the vehicle must deal with various Functional Insufficiencies (FIs). Thus-far, there is no widely accepted functional architecture that maximizes the availability of autonomy and ensures safety in complex vehicle operational design domains. In this paper, we present a survey of existing methods that strive to preven…
▽ More
To enable highly automated vehicles where the driver is no longer a safety backup, the vehicle must deal with various Functional Insufficiencies (FIs). Thus-far, there is no widely accepted functional architecture that maximizes the availability of autonomy and ensures safety in complex vehicle operational design domains. In this paper, we present a survey of existing methods that strive to prevent or handle FIs. We observe that current design-time methods of preventing FIs lack completeness guarantees. Complementary solutions for on-line handling cannot suitably increase safety without seriously impacting availability of journey continuing autonomous functionality. To fill this gap, we propose the Safety Shell, a scalable multi-channel architecture and arbitration design, built upon preexisting functional safety redundant channel architectures. We compare this novel approach to existing architectures using numerical case studies. The results show that the Safety Shell architecture allows the automated vehicle to be as safe or safer compared to alternatives, while simultaneously improving availability of vehicle autonomy, thereby increasing the possible coverage of on-line functional insufficiency handling.
△ Less
Submitted 21 November, 2023; v1 submitted 20 October, 2023;
originally announced November 2023.
-
Overcoming the Fear of the Dark: Occlusion-Aware Model-Predictive Planning for Automated Vehicles Using Risk Fields
Authors:
Chris van der Ploeg,
Truls Nyberg,
José Manuel Gaspar Sánchez,
Emilia Silvas,
Nathan van de Wouw
Abstract:
As vehicle automation advances, motion planning algorithms face escalating challenges in achieving safe and efficient navigation. Existing Advanced Driver Assistance Systems (ADAS) primarily focus on basic tasks, leaving unexpected scenarios for human intervention, which can be error-prone. Motion planning approaches for higher levels of automation in the state-of-the-art are primarily oriented to…
▽ More
As vehicle automation advances, motion planning algorithms face escalating challenges in achieving safe and efficient navigation. Existing Advanced Driver Assistance Systems (ADAS) primarily focus on basic tasks, leaving unexpected scenarios for human intervention, which can be error-prone. Motion planning approaches for higher levels of automation in the state-of-the-art are primarily oriented toward the use of risk- or anti-collision constraints, using over-approximates of the shapes and sizes of other road users to prevent collisions. These methods however suffer from conservative behavior and the risk of infeasibility in high-risk initial conditions. In contrast, our work introduces a novel multi-objective trajectory generation approach. We propose an innovative method for constructing risk fields that accommodates diverse entity shapes and sizes, which allows us to also account for the presence of potentially occluded objects. This methodology is integrated into an occlusion-aware trajectory generator, enabling dynamic and safe maneuvering through intricate environments while anticipating (potentially hidden) road users and traveling along the infrastructure toward a specific goal. Through theoretical underpinnings and simulations, we validate the effectiveness of our approach. This paper bridges crucial gaps in motion planning for automated vehicles, offering a pathway toward safer and more adaptable autonomous navigation in complex urban contexts.
△ Less
Submitted 27 September, 2023;
originally announced September 2023.
-
Robustness Benchmark of Road User Trajectory Prediction Models for Automated Driving
Authors:
Manuel Muñoz Sánchez,
Emilia Silvas,
Jos Elfring,
René van de Molengraft
Abstract:
Accurate and robust trajectory predictions of road users are needed to enable safe automated driving. To do this, machine learning models are often used, which can show erratic behavior when presented with previously unseen inputs. In this work, two environment-aware models (MotionCNN and MultiPath++) and two common baselines (Constant Velocity and an LSTM) are benchmarked for robustness against v…
▽ More
Accurate and robust trajectory predictions of road users are needed to enable safe automated driving. To do this, machine learning models are often used, which can show erratic behavior when presented with previously unseen inputs. In this work, two environment-aware models (MotionCNN and MultiPath++) and two common baselines (Constant Velocity and an LSTM) are benchmarked for robustness against various perturbations that simulate functional insufficiencies observed during model deployment in a vehicle: unavailability of road information, late detections, and noise. Results show significant performance degradation under the presence of these perturbations, with errors increasing up to +1444.8\% in commonly used trajectory prediction evaluation metrics. Training the models with similar perturbations effectively reduces performance degradation, with error increases of up to +87.5\%. We argue that despite being an effective mitigation strategy, data augmentation through perturbations during training does not guarantee robustness towards unforeseen perturbations, since identification of all possible on-road complications is unfeasible. Furthermore, degrading the inputs sometimes leads to more accurate predictions, suggesting that the models are unable to learn the true relationships between the different elements in the data.
△ Less
Submitted 4 April, 2023;
originally announced April 2023.
-
Optimization-based Fault Mitigation for Safe Automated Driving
Authors:
Niels Lodder,
Chris van der Ploeg,
Laura Ferranti,
Emilia Silvas
Abstract:
With increased developments and interest in cooperative driving and higher levels of automation (SAE level 3+), the need for safety systems that are capable to monitor system health and maintain safe operations in faulty scenarios is increasing. A variety of faults or failures could occur, and there exists a high variety of ways to respond to such events. Once a fault or failure is detected, there…
▽ More
With increased developments and interest in cooperative driving and higher levels of automation (SAE level 3+), the need for safety systems that are capable to monitor system health and maintain safe operations in faulty scenarios is increasing. A variety of faults or failures could occur, and there exists a high variety of ways to respond to such events. Once a fault or failure is detected, there is a need to classify its severity and decide on appropriate and safe mitigating actions. To provide a solution to this mitigation challenge, in this paper a functional-safety architecture is proposed and an optimization-based mitigation algorithm is introduced. This algorithm uses nonlinear model predictive control (NMPC) to bring a vehicle, suffering from a severe fault, such as a power steering failure, to a safe-state. The internal model of the NMPC uses the information from the fault detection, isolation and identification to optimize the tracking performance of the controller, showcasing the need of the proposed architecture. Given a string of ACC vehicles, our results demonstrate a variety of tactical decision-making approaches that a fault-affected vehicle could employ to manage any faults. Furthermore, we show the potential for improving the safety of the affected vehicle as well as the effect of these approaches on the duration of the manoeuvre.
△ Less
Submitted 31 March, 2023;
originally announced March 2023.
-
Scenario-based Evaluation of Prediction Models for Automated Vehicles
Authors:
Manuel Muñoz Sánchez,
Jos Elfring,
Emilia Silvas,
René van de Molengraft
Abstract:
To operate safely, an automated vehicle (AV) must anticipate how the environment around it will evolve. For that purpose, it is important to know which prediction models are most appropriate for every situation. Currently, assessment of prediction models is often performed over a set of trajectories without distinction of the type of movement they capture, resulting in the inability to determine t…
▽ More
To operate safely, an automated vehicle (AV) must anticipate how the environment around it will evolve. For that purpose, it is important to know which prediction models are most appropriate for every situation. Currently, assessment of prediction models is often performed over a set of trajectories without distinction of the type of movement they capture, resulting in the inability to determine the suitability of each model for different situations. In this work we illustrate how standardized evaluation methods result in wrong conclusions regarding a model's predictive capabilities, preventing a clear assessment of prediction models and potentially leading to dangerous on-road situations. We argue that following evaluation practices in safety assessment for AVs, assessment of prediction models should be performed in a scenario-based fashion. To encourage scenario-based assessment of prediction models and illustrate the dangers of improper assessment, we categorize trajectories of the Waymo Open Motion dataset according to the type of movement they capture. Next, three different models are thoroughly evaluated for different trajectory types and prediction horizons. Results show that common evaluation methods are insufficient and the assessment should be performed depending on the application in which the model will operate.
△ Less
Submitted 11 October, 2022;
originally announced October 2022.
-
Informed sampling-based trajectory planner for automated driving in dynamic urban environments
Authors:
Robin Smit,
Chris van der Ploeg,
Arjan Teerhuis,
Emilia Silvas
Abstract:
The urban environment is amongst the most difficult domains for autonomous vehicles. The vehicle must be able to plan a safe route on challenging road layouts, in the presence of various dynamic traffic participants such as vehicles, cyclists and pedestrians and in various environmental conditions. The challenge remains to have motion planners that are computationally fast and that account for fut…
▽ More
The urban environment is amongst the most difficult domains for autonomous vehicles. The vehicle must be able to plan a safe route on challenging road layouts, in the presence of various dynamic traffic participants such as vehicles, cyclists and pedestrians and in various environmental conditions. The challenge remains to have motion planners that are computationally fast and that account for future movements of other road users proactively. This paper describes an computationally efficient sampling-based trajectory planner for safe and comfortable driving in urban environments. The planner improves the Stable-Sparse-RRT algorithm by adding initial exploration branches to the search tree based on road layout information and reiterating the previous solution. Furthermore, the trajectory planner accounts for the predicted motion of other traffic participants to allow for safe driving in urban traffic. Simulation studies show that the planner is capable of planning collision-free, comfortable trajectories in several urban traffic scenarios. Adding the domain-knowledge-based exploration branches increases the efficiency of exploration of highly interesting areas, thereby increasing the overall planning performance.
△ Less
Submitted 5 October, 2022;
originally announced October 2022.
-
Long Horizon Risk-Averse Motion Planning: a Model-Predictive Approach
Authors:
Chris van der Ploeg,
Robin Smit,
Arjan Teerhuis,
Emilia Silvas
Abstract:
Developing safe automated vehicles that can be proactive, safe, and comfortable in mixed traffic requires improved planning methods that are risk-averse and that account for predictions of the motion of other road users. To consider these criteria, in this article, we propose a non-linear model-predictive trajectory generator scheme, which couples the longitudinal and lateral motion of the vehicle…
▽ More
Developing safe automated vehicles that can be proactive, safe, and comfortable in mixed traffic requires improved planning methods that are risk-averse and that account for predictions of the motion of other road users. To consider these criteria, in this article, we propose a non-linear model-predictive trajectory generator scheme, which couples the longitudinal and lateral motion of the vehicle to steer the vehicle with minimal risk, while progressing towards the goal state. The proposed method takes into account the infrastructure, surrounding objects, and predictions of the objects' state through artificial potential-based risk fields included in the cost function of the model-predictive control (MPC) problem. This trajectory generator enables anticipatory maneuvers, i.e., mitigating risk far before any safety-critical intervention would be necessary. The method is proven in several case studies representing both highways- and urban situations. The results show the safe and efficient implementation of the MPC trajectory generator while proving its real-time applicability.
△ Less
Submitted 15 September, 2022;
originally announced September 2022.
-
Anomaly Detection from Cyber Threats via Infrastructure to Automated Vehicle
Authors:
Chris van der Ploeg,
Robin Smit,
Alexis Siagkris-Lekkos,
Frank Benders,
Emilia Silvas
Abstract:
Using Infrastructure-to-Vehicle (I2V) information can be of great benefit when driving autonomously in high-density traffic situations with limited visibility, since the sensing capabilities of the vehicle are enhanced by external sensors. In this research, a method is introduced to increase the vehicle's self-awareness in intersections for one of the largest foreseen challenges when using I2V com…
▽ More
Using Infrastructure-to-Vehicle (I2V) information can be of great benefit when driving autonomously in high-density traffic situations with limited visibility, since the sensing capabilities of the vehicle are enhanced by external sensors. In this research, a method is introduced to increase the vehicle's self-awareness in intersections for one of the largest foreseen challenges when using I2V communication: cyber security. The introduced anomaly detection algorithm, running on the automated vehicle, assesses the health of the I2V communication against multiple cyber security attacks. The analysis is done in a simulation environment, using cyber-attack scenarios from the Secredas Project (Cyber Security for Cross Domain Reliable Dependable Automated Systems) and provides insights into the limitations the vehicle has when facing I2V cyber attacks of different types and amplitudes and when sensor redundancy is lost. The results demonstrate that anomalies injected can be robustly detected and mitigated by the autonomous vehicle, allowing it to react more safely and comfortably and maintaining correct object tracking in intersections.
△ Less
Submitted 23 April, 2021;
originally announced April 2021.
-
Real-time Fault Estimation for a Class of Discrete-Time Linear Parameter-Varying Systems
Authors:
Chris van der Ploeg,
Emilia Silvas,
Nathan van de Wouw,
Peyman Mohajerin Esfahani
Abstract:
Estimating and detecting faults is crucial in ensuring safe and efficient automated systems. In the presence of disturbances, noise or varying system dynamics, such estimation is even more challenging. To address this challenge, this article proposes a novel filter to estimate multiple fault signals for a class of discrete-time linear parameter-varying (LPV) systems. The design of such a filter is…
▽ More
Estimating and detecting faults is crucial in ensuring safe and efficient automated systems. In the presence of disturbances, noise or varying system dynamics, such estimation is even more challenging. To address this challenge, this article proposes a novel filter to estimate multiple fault signals for a class of discrete-time linear parameter-varying (LPV) systems. The design of such a filter is formulated as an optimization problem and is solved recursively, while the system dynamics may vary over time. Conditions for existence and detectability of the fault are introduced and the problem is formulated and solved using the quadratic programming framework. We further propose an approximate scheme that can be arbitrarily precise while it enjoys an analytical solution, which supports real-time implementation. The method is illustrated and validated on an automated vehicle's lateral dynamics, which is a practically relevant example for LPV systems. The results show that the estimation filter can decouple unknown disturbances and known or measurable parameter variations in the dynamics while estimating the unknown fault.
△ Less
Submitted 10 December, 2021; v1 submitted 23 April, 2021;
originally announced April 2021.
