-
Checklist to Define the Identification of TP, FP, and FN Object Detections in Automated Driving
Authors:
Michael Hoss
Abstract:
The object perception of automated driving systems must pass quality and robustness tests before a safe deployment. Such tests typically identify true positive (TP), false-positive (FP), and false-negative (FN) detections and aggregate them to metrics. Since the literature seems to be lacking a comprehensive way to define the identification of TPs/FPs/FNs, this paper provides a checklist of releva…
▽ More
The object perception of automated driving systems must pass quality and robustness tests before a safe deployment. Such tests typically identify true positive (TP), false-positive (FP), and false-negative (FN) detections and aggregate them to metrics. Since the literature seems to be lacking a comprehensive way to define the identification of TPs/FPs/FNs, this paper provides a checklist of relevant functional aspects and implementation details. Besides labeling policies of the test set, we cover areas of vision, occlusion handling, safety-relevant areas, matching criteria, temporal and probabilistic issues, and further aspects. Even though the checklist cannot be fully formalized, it can help practitioners minimize the ambiguity of their tests, which, in turn, makes statements on object perception more reliable and comparable.
△ Less
Submitted 18 September, 2024; v1 submitted 14 August, 2023;
originally announced August 2023.
-
A Review of Testing Object-Based Environment Perception for Safe Automated Driving
Authors:
Michael Hoss,
Maike Scholtes,
Lutz Eckstein
Abstract:
Safety assurance of automated driving systems must consider uncertain environment perception. This paper reviews literature addressing how perception testing is realized as part of safety assurance. We focus on testing for verification and validation purposes at the interface between perception and planning, and structure our analysis along the three axes 1) test criteria and metrics, 2) test scen…
▽ More
Safety assurance of automated driving systems must consider uncertain environment perception. This paper reviews literature addressing how perception testing is realized as part of safety assurance. We focus on testing for verification and validation purposes at the interface between perception and planning, and structure our analysis along the three axes 1) test criteria and metrics, 2) test scenarios, and 3) reference data. Furthermore, the analyzed literature includes related safety standards, safety-independent perception algorithm benchmarking, and sensor modeling. We find that the realization of safety-aware perception testing remains an open issue since challenges concerning the three testing axes and their interdependencies currently do not appear to be sufficiently solved.
△ Less
Submitted 16 February, 2021;
originally announced February 2021.
-
6-Layer Model for a Structured Description and Categorization of Urban Traffic and Environment
Authors:
Maike Scholtes,
Lukas Westhofen,
Lara Ruth Turner,
Katrin Lotto,
Michael Schuldes,
Hendrik Weber,
Nicolas Wagener,
Christian Neurohr,
Martin Bollmann,
Franziska Körtke,
Johannes Hiller,
Michael Hoss,
Julian Bock,
Lutz Eckstein
Abstract:
Verification and validation of automated driving functions impose large challenges. Currently, scenario-based approaches are investigated in research and industry, aiming at a reduction of testing efforts by specifying safety relevant scenarios. To define those scenarios and operate in a complex real-world design domain, a structured description of the environment is needed. Within the PEGASUS res…
▽ More
Verification and validation of automated driving functions impose large challenges. Currently, scenario-based approaches are investigated in research and industry, aiming at a reduction of testing efforts by specifying safety relevant scenarios. To define those scenarios and operate in a complex real-world design domain, a structured description of the environment is needed. Within the PEGASUS research project, the 6-Layer Model (6LM) was introduced for the description of highway scenarios. This paper refines the 6LM and extends it to urban traffic and environment. As defined in PEGASUS, the 6LM provides the possibility to categorize the environment and, therefore, functions as a structured basis for subsequent scenario description. The model enables a structured description and categorization of the general environment, without incorporating any knowledge or anticipating any functions of actors. Beyond that, there is a variety of other applications of the 6LM, which are elaborated in this paper. The 6LM includes a description of the road network and traffic guidance objects, roadside structures, temporary modifications of the former, dynamic objects, environmental conditions and digital information. The work at hand specifies each layer by categorizing its items. Guidelines are formulated and explanatory examples are given to standardize the application of the model for an objective environment description. In contrast to previous publications, the model and its design are described in far more detail. Finally, the holistic description of the 6LM presented includes remarks on possible future work when expanding the concept to machine perception aspects.
△ Less
Submitted 2 February, 2021; v1 submitted 9 December, 2020;
originally announced December 2020.
