Showing 1–28 of 28 results for author: Bogdoll, D

Mcity Data Engine: Iterative Model Improvement Through Open-Vocabulary Data Selection

Authors: Daniel Bogdoll, Rajanikant Patnaik Ananta, Abeyankar Giridharan, Isabel Moore, Gregory Stevens, Henry X. Liu

Abstract: With an ever-increasing availability of data, it has become more and more challenging to select and label appropriate samples for the training of machine learning models. It is especially difficult to detect long-tail classes of interest in large amounts of unlabeled data. This holds especially true for Intelligent Transportation Systems (ITS), where vehicle fleets and roadside perception systems… ▽ More With an ever-increasing availability of data, it has become more and more challenging to select and label appropriate samples for the training of machine learning models. It is especially difficult to detect long-tail classes of interest in large amounts of unlabeled data. This holds especially true for Intelligent Transportation Systems (ITS), where vehicle fleets and roadside perception systems generate an abundance of raw data. While industrial, proprietary data engines for such iterative data selection and model training processes exist, researchers and the open-source community suffer from a lack of an openly available system. We present the Mcity Data Engine, which provides modules for the complete data-based development cycle, beginning at the data acquisition phase and ending at the model deployment stage. The Mcity Data Engine focuses on rare and novel classes through an open-vocabulary data selection process. All code is publicly available on GitHub under an MIT license: https://github.com/mcity/mcity_data_engine △ Less

Submitted 30 April, 2025; originally announced April 2025.

Label-Free Model Failure Detection for Lidar-based Point Cloud Segmentation

Authors: Daniel Bogdoll, Finn Sartoris, Vincent Geppert, Svetlana Pavlitska, J. Marius Zöllner

Abstract: Autonomous vehicles drive millions of miles on the road each year. Under such circumstances, deployed machine learning models are prone to failure both in seemingly normal situations and in the presence of outliers. However, in the training phase, they are only evaluated on small validation and test sets, which are unable to reveal model failures due to their limited scenario coverage. While it is… ▽ More Autonomous vehicles drive millions of miles on the road each year. Under such circumstances, deployed machine learning models are prone to failure both in seemingly normal situations and in the presence of outliers. However, in the training phase, they are only evaluated on small validation and test sets, which are unable to reveal model failures due to their limited scenario coverage. While it is difficult and expensive to acquire large and representative labeled datasets for evaluation, large-scale unlabeled datasets are typically available. In this work, we introduce label-free model failure detection for lidar-based point cloud segmentation, taking advantage of the abundance of unlabeled data available. We leverage different data characteristics by training a supervised and self-supervised stream for the same task to detect failure modes. We perform a large-scale qualitative analysis and present LidarCODA, the first publicly available dataset with labeled anomalies in real-world lidar data, for an extensive quantitative analysis. △ Less

Submitted 24 April, 2025; v1 submitted 19 July, 2024; originally announced July 2024.

Comments: Daniel Bogdoll, Finn Sartoris, and Vincent Geppert contributed equally. Accepted for publication at IV 2025

Hybrid Video Anomaly Detection for Anomalous Scenarios in Autonomous Driving

Authors: Daniel Bogdoll, Jan Imhof, Tim Joseph, Svetlana Pavlitska, J. Marius Zöllner

Abstract: In autonomous driving, the most challenging scenarios can only be detected within their temporal context. Most video anomaly detection approaches focus either on surveillance or traffic accidents, which are only a subfield of autonomous driving. We present HF$^2$-VAD$_{AD}$, a variation of the HF$^2$-VAD surveillance video anomaly detection method for autonomous driving. We learn a representation… ▽ More In autonomous driving, the most challenging scenarios can only be detected within their temporal context. Most video anomaly detection approaches focus either on surveillance or traffic accidents, which are only a subfield of autonomous driving. We present HF$^2$-VAD$_{AD}$, a variation of the HF$^2$-VAD surveillance video anomaly detection method for autonomous driving. We learn a representation of normality from a vehicle's ego perspective and evaluate pixel-wise anomaly detections in rare and critical scenarios. △ Less

Submitted 28 April, 2025; v1 submitted 10 June, 2024; originally announced June 2024.

Comments: Daniel Bogdoll and Jan Imhof contributed equally. Accepted for publication at BMVC 2024 RROW workshop. Won Best Paper Award

UMAD: Unsupervised Mask-Level Anomaly Detection for Autonomous Driving

Authors: Daniel Bogdoll, Noël Ollick, Tim Joseph, Svetlana Pavlitska, J. Marius Zöllner

Abstract: Dealing with atypical traffic scenarios remains a challenging task in autonomous driving. However, most anomaly detection approaches cannot be trained on raw sensor data but require exposure to outlier data and powerful semantic segmentation models trained in a supervised fashion. This limits the representation of normality to labeled data, which does not scale well. In this work, we revisit unsup… ▽ More Dealing with atypical traffic scenarios remains a challenging task in autonomous driving. However, most anomaly detection approaches cannot be trained on raw sensor data but require exposure to outlier data and powerful semantic segmentation models trained in a supervised fashion. This limits the representation of normality to labeled data, which does not scale well. In this work, we revisit unsupervised anomaly detection and present UMAD, leveraging generative world models and unsupervised image segmentation. Our method outperforms state-of-the-art unsupervised anomaly detection. △ Less

Submitted 30 September, 2024; v1 submitted 10 June, 2024; originally announced June 2024.

Comments: Daniel Bogdoll and Noël Ollick contributed equally. Accepted for publication at BMVC 2024 RROW workshop

AnoVox: A Benchmark for Multimodal Anomaly Detection in Autonomous Driving

Authors: Daniel Bogdoll, Iramm Hamdard, Lukas Namgyu Rößler, Felix Geisler, Muhammed Bayram, Felix Wang, Jan Imhof, Miguel de Campos, Anushervon Tabarov, Yitian Yang, Hanno Gottschalk, J. Marius Zöllner

Abstract: The scale-up of autonomous vehicles depends heavily on their ability to deal with anomalies, such as rare objects on the road. In order to handle such situations, it is necessary to detect anomalies in the first place. Anomaly detection for autonomous driving has made great progress in the past years but suffers from poorly designed benchmarks with a strong focus on camera data. In this work, we p… ▽ More The scale-up of autonomous vehicles depends heavily on their ability to deal with anomalies, such as rare objects on the road. In order to handle such situations, it is necessary to detect anomalies in the first place. Anomaly detection for autonomous driving has made great progress in the past years but suffers from poorly designed benchmarks with a strong focus on camera data. In this work, we propose AnoVox, the largest benchmark for ANOmaly detection in autonomous driving to date. AnoVox incorporates large-scale multimodal sensor data and spatial VOXel ground truth, allowing for the comparison of methods independent of their used sensor. We propose a formal definition of normality and provide a compliant training dataset. AnoVox is the first benchmark to contain both content and temporal anomalies. △ Less

Submitted 26 September, 2024; v1 submitted 13 May, 2024; originally announced May 2024.

Comments: Daniel Bogdoll, Iramm Hamdard, and Lukas Namgyu Rößler contributed equally. Accepted for publication at ECCV 2024 W-CODA workshop

One Stack to Rule them All: To Drive Automated Vehicles, and Reach for the 4th level

Authors: Sven Ochs, Jens Doll, Daniel Grimm, Tobias Fleck, Marc Heinrich, Stefan Orf, Albert Schotschneider, Helen Gremmelmaier, Rupert Polley, Svetlana Pavlitska, Maximilian Zipfl, Helen Schneider, Ferdinand Mütsch, Daniel Bogdoll, Florian Kuhnt, Philip Schörner, Marc René Zofka, J. Marius Zöllner

Abstract: Most automated driving functions are designed for a specific task or vehicle. Most often, the underlying architecture is fixed to specific algorithms to increase performance. Therefore, it is not possible to deploy new modules and algorithms easily. In this paper, we present our automated driving stack which combines both scalability and adaptability. Due to the modular design, our stack allows fo… ▽ More Most automated driving functions are designed for a specific task or vehicle. Most often, the underlying architecture is fixed to specific algorithms to increase performance. Therefore, it is not possible to deploy new modules and algorithms easily. In this paper, we present our automated driving stack which combines both scalability and adaptability. Due to the modular design, our stack allows for a fast integration and testing of novel and state-of-the-art research approaches. Furthermore, it is flexible to be used for our different testing vehicles, including modified EasyMile EZ10 shuttles and different passenger cars. These vehicles differ in multiple ways, e.g. sensor setups, control systems, maximum speed, or steering angle limitations. Finally, our stack is deployed in real world environments, including passenger transport in urban areas. Our stack includes all components needed for operating an autonomous vehicle, including localization, perception, planning, controller, and additional safety modules. Our stack is developed, tested, and evaluated in real world traffic in multiple test sites, including the Test Area Autonomous Driving Baden-Württemberg. △ Less

Submitted 3 April, 2024; originally announced April 2024.

Informed Reinforcement Learning for Situation-Aware Traffic Rule Exceptions

Authors: Daniel Bogdoll, Jing Qin, Moritz Nekolla, Ahmed Abouelazm, Tim Joseph, J. Marius Zöllner

Abstract: Reinforcement Learning is a highly active research field with promising advancements. In the field of autonomous driving, however, often very simple scenarios are being examined. Common approaches use non-interpretable control commands as the action space and unstructured reward designs which lack structure. In this work, we introduce Informed Reinforcement Learning, where a structured rulebook is… ▽ More Reinforcement Learning is a highly active research field with promising advancements. In the field of autonomous driving, however, often very simple scenarios are being examined. Common approaches use non-interpretable control commands as the action space and unstructured reward designs which lack structure. In this work, we introduce Informed Reinforcement Learning, where a structured rulebook is integrated as a knowledge source. We learn trajectories and asses them with a situation-aware reward design, leading to a dynamic reward which allows the agent to learn situations which require controlled traffic rule exceptions. Our method is applicable to arbitrary RL models. We successfully demonstrate high completion rates of complex scenarios with recent model-based agents. △ Less

Submitted 12 June, 2024; v1 submitted 6 February, 2024; originally announced February 2024.

Comments: Daniel Bogdoll and Jing Qin contributed equally. Accepted for publication at ICRA 2024

On The Impact of Replacing Private Cars with Autonomous Shuttles: An Agent-Based Approach

Authors: Daniel Bogdoll, Louis Karsch, Jennifer Amritzer, J. Marius Zöllner

Abstract: The European Green Deal aims to achieve climate neutrality by 2050, which demands improved emissions efficiency from the transportation industry. This study uses an agent-based simulation to analyze the sustainability impacts of shared autonomous shuttles. We forecast travel demands for 2050 and simulate regulatory interventions in the form of replacing private cars with a fleet of shared autonomo… ▽ More The European Green Deal aims to achieve climate neutrality by 2050, which demands improved emissions efficiency from the transportation industry. This study uses an agent-based simulation to analyze the sustainability impacts of shared autonomous shuttles. We forecast travel demands for 2050 and simulate regulatory interventions in the form of replacing private cars with a fleet of shared autonomous shuttles in specific areas. We derive driving-related emissions, energy consumption, and non-driving-related emissions to calculate life-cycle emissions. We observe reduced life-cycle emissions from 0.4% to 9.6% and reduced energy consumption from 1.5% to 12.2%. △ Less

Submitted 19 January, 2024; v1 submitted 23 November, 2023; originally announced November 2023.

Comments: Daniel Bogdoll and Louis Karsch contributed equally. Accepted for publication at FISTS 2024

MUVO: A Multimodal Generative World Model for Autonomous Driving with Geometric Representations

Authors: Daniel Bogdoll, Yitian Yang, Tim Joseph, Melih Yazgan, J. Marius Zöllner

Abstract: World models for autonomous driving have the potential to dramatically improve the reasoning capabilities of today's systems. However, most works focus on camera data, with only a few that leverage lidar data or combine both to better represent autonomous vehicle sensor setups. In addition, raw sensor predictions are less actionable than 3D occupancy predictions, but there are no works examining t… ▽ More World models for autonomous driving have the potential to dramatically improve the reasoning capabilities of today's systems. However, most works focus on camera data, with only a few that leverage lidar data or combine both to better represent autonomous vehicle sensor setups. In addition, raw sensor predictions are less actionable than 3D occupancy predictions, but there are no works examining the effects of combining both multimodal sensor data and 3D occupancy prediction. In this work, we perform a set of experiments with a MUltimodal World Model with Geometric VOxel representations (MUVO) to evaluate different sensor fusion strategies to better understand the effects on sensor data prediction. We also analyze potential weaknesses of current sensor fusion approaches and examine the benefits of additionally predicting 3D occupancy. △ Less

Submitted 24 April, 2025; v1 submitted 20 November, 2023; originally announced November 2023.

Comments: Daniel Bogdoll and Yitian Yang contributed equally. Accepted for publication at IV 2025

Conditioning Latent-Space Clusters for Real-World Anomaly Classification

Authors: Daniel Bogdoll, Svetlana Pavlitska, Simon Klaus, J. Marius Zöllner

Abstract: Anomalies in the domain of autonomous driving are a major hindrance to the large-scale deployment of autonomous vehicles. In this work, we focus on high-resolution camera data from urban scenes that include anomalies of various types and sizes. Based on a Variational Autoencoder, we condition its latent space to classify samples as either normal data or anomalies. In order to emphasize especially… ▽ More Anomalies in the domain of autonomous driving are a major hindrance to the large-scale deployment of autonomous vehicles. In this work, we focus on high-resolution camera data from urban scenes that include anomalies of various types and sizes. Based on a Variational Autoencoder, we condition its latent space to classify samples as either normal data or anomalies. In order to emphasize especially small anomalies, we perform experiments where we provide the VAE with a discrepancy map as an additional input, evaluating its impact on the detection performance. Our method separates normal data and anomalies into isolated clusters while still reconstructing high-quality images, leading to meaningful latent representations. △ Less

Submitted 18 September, 2023; originally announced September 2023.

Comments: Daniel Bogdoll, Svetlana Pavlitska, and Simon Klaus contributed equally. Accepted for publication at SSCI 2023

Exploring the Potential of World Models for Anomaly Detection in Autonomous Driving

Authors: Daniel Bogdoll, Lukas Bosch, Tim Joseph, Helen Gremmelmaier, Yitian Yang, J. Marius Zöllner

Abstract: In recent years there have been remarkable advancements in autonomous driving. While autonomous vehicles demonstrate high performance in closed-set conditions, they encounter difficulties when confronted with unexpected situations. At the same time, world models emerged in the field of model-based reinforcement learning as a way to enable agents to predict the future depending on potential actions… ▽ More In recent years there have been remarkable advancements in autonomous driving. While autonomous vehicles demonstrate high performance in closed-set conditions, they encounter difficulties when confronted with unexpected situations. At the same time, world models emerged in the field of model-based reinforcement learning as a way to enable agents to predict the future depending on potential actions. This led to outstanding results in sparse reward and complex control tasks. This work provides an overview of how world models can be leveraged to perform anomaly detection in the domain of autonomous driving. We provide a characterization of world models and relate individual components to previous works in anomaly detection to facilitate further research in the field. △ Less

Submitted 18 September, 2023; v1 submitted 10 August, 2023; originally announced August 2023.

Comments: Accepted for publication at SSCI 2023

From Model-Based to Data-Driven Simulation: Challenges and Trends in Autonomous Driving

Authors: Ferdinand Mütsch, Helen Gremmelmaier, Nicolas Becker, Daniel Bogdoll, Marc René Zofka, J. Marius Zöllner

Abstract: Simulation is an integral part in the process of developing autonomous vehicles and advantageous for training, validation, and verification of driving functions. Even though simulations come with a series of benefits compared to real-world experiments, various challenges still prevent virtual testing from entirely replacing physical test-drives. Our work provides an overview of these challenges wi… ▽ More Simulation is an integral part in the process of developing autonomous vehicles and advantageous for training, validation, and verification of driving functions. Even though simulations come with a series of benefits compared to real-world experiments, various challenges still prevent virtual testing from entirely replacing physical test-drives. Our work provides an overview of these challenges with regard to different aspects and types of simulation and subsumes current trends to overcome them. We cover aspects around perception-, behavior- and content-realism as well as general hurdles in the domain of simulation. Among others, we observe a trend of data-driven, generative approaches and high-fidelity data synthesis to increasingly replace model-based simulation. △ Less

Submitted 31 July, 2023; v1 submitted 23 May, 2023; originally announced May 2023.

Comments: Ferdinand Mütsch, Helen Gremmelmaier, and Nicolas Becker contributed equally. Accepted for publication at CVPR 2023 VCAD workshop

Perception Datasets for Anomaly Detection in Autonomous Driving: A Survey

Authors: Daniel Bogdoll, Svenja Uhlemeyer, Kamil Kowol, J. Marius Zöllner

Abstract: Deep neural networks (DNN) which are employed in perception systems for autonomous driving require a huge amount of data to train on, as they must reliably achieve high performance in all kinds of situations. However, these DNN are usually restricted to a closed set of semantic classes available in their training data, and are therefore unreliable when confronted with previously unseen instances.… ▽ More Deep neural networks (DNN) which are employed in perception systems for autonomous driving require a huge amount of data to train on, as they must reliably achieve high performance in all kinds of situations. However, these DNN are usually restricted to a closed set of semantic classes available in their training data, and are therefore unreliable when confronted with previously unseen instances. Thus, multiple perception datasets have been created for the evaluation of anomaly detection methods, which can be categorized into three groups: real anomalies in real-world, synthetic anomalies augmented into real-world and completely synthetic scenes. This survey provides a structured and, to the best of our knowledge, complete overview and comparison of perception datasets for anomaly detection in autonomous driving. Each chapter provides information about tasks and ground truth, context information, and licenses. Additionally, we discuss current weaknesses and gaps in existing datasets to underline the importance of developing further data. △ Less

Submitted 31 March, 2023; v1 submitted 6 February, 2023; originally announced February 2023.

Comments: Accepted for publication at IV 2023

Impact, Attention, Influence: Early Assessment of Autonomous Driving Datasets

Authors: Daniel Bogdoll, Jonas Hendl, Felix Schreyer, Nishanth Gowda, Michael Färber, J. Marius Zöllner

Abstract: Autonomous Driving (AD), the area of robotics with the greatest potential impact on society, has gained a lot of momentum in the last decade. As a result of this, the number of datasets in AD has increased rapidly. Creators and users of datasets can benefit from a better understanding of developments in the field. While scientometric analysis has been conducted in other fields, it rarely revolves… ▽ More Autonomous Driving (AD), the area of robotics with the greatest potential impact on society, has gained a lot of momentum in the last decade. As a result of this, the number of datasets in AD has increased rapidly. Creators and users of datasets can benefit from a better understanding of developments in the field. While scientometric analysis has been conducted in other fields, it rarely revolves around datasets. Thus, the impact, attention, and influence of datasets on autonomous driving remains a rarely investigated field. In this work, we provide a scientometric analysis for over 200 datasets in AD. We perform a rigorous evaluation of relations between available metadata and citation counts based on linear regression. Subsequently, we propose an Influence Score to assess a dataset already early on without the need for a track-record of citations, which is only available with a certain delay. △ Less

Submitted 31 March, 2023; v1 submitted 5 January, 2023; originally announced January 2023.

Comments: Daniel Bogdoll and Jonas Hendl contributed equally. Accepted for publication at ICCRE 2023

One Ontology to Rule Them All: Corner Case Scenarios for Autonomous Driving

Authors: Daniel Bogdoll, Stefani Guneshka, J. Marius Zöllner

Abstract: The core obstacle towards a large-scale deployment of autonomous vehicles currently lies in the long tail of rare events. These are extremely challenging since they do not occur often in the utilized training data for deep neural networks. To tackle this problem, we propose the generation of additional synthetic training data, covering a wide variety of corner case scenarios. As ontologies can rep… ▽ More The core obstacle towards a large-scale deployment of autonomous vehicles currently lies in the long tail of rare events. These are extremely challenging since they do not occur often in the utilized training data for deep neural networks. To tackle this problem, we propose the generation of additional synthetic training data, covering a wide variety of corner case scenarios. As ontologies can represent human expert knowledge while enabling computational processing, we use them to describe scenarios. Our proposed master ontology is capable to model scenarios from all common corner case categories found in the literature. From this one master ontology, arbitrary scenario-describing ontologies can be derived. In an automated fashion, these can be converted into the OpenSCENARIO format and subsequently executed in simulation. This way, also challenging test and evaluation scenarios can be generated. △ Less

Submitted 4 October, 2022; v1 submitted 1 September, 2022; originally announced September 2022.

Comments: Daniel Bogdoll and Stefani Guneshka contributed equally. Accepted for publication at ECCV 2022 SAIAD workshop

DLCSS: Dynamic Longest Common Subsequences

Authors: Daniel Bogdoll, Jonas Rauch, J. Marius Zöllner

Abstract: Autonomous driving is a key technology towards a brighter, more sustainable future. To enable such a future, it is necessary to utilize autonomous vehicles in shared mobility models. However, to evaluate, whether two or more route requests have the potential for a shared ride, is a compute-intensive task, if done by rerouting. In this work, we propose the Dynamic Longest Common Subsequences algori… ▽ More Autonomous driving is a key technology towards a brighter, more sustainable future. To enable such a future, it is necessary to utilize autonomous vehicles in shared mobility models. However, to evaluate, whether two or more route requests have the potential for a shared ride, is a compute-intensive task, if done by rerouting. In this work, we propose the Dynamic Longest Common Subsequences algorithm for fast and cost-efficient comparison of two routes for their compatibility, dynamically only incorporating parts of the routes which are suited for a shared trip. Based on this, one can also estimate, how many autonomous vehicles might be necessary to fulfill the local mobility demands. This can help providers to estimate the necessary fleet sizes, policymakers to better understand mobility patterns and cities to scale necessary infrastructure. △ Less

Submitted 1 September, 2022; v1 submitted 13 July, 2022; originally announced July 2022.

Comments: Accepted for publication at ICECCME 2022

Experiments on Anomaly Detection in Autonomous Driving by Forward-Backward Style Transfers

Authors: Daniel Bogdoll, Meng Zhang, Maximilian Nitsche, J. Marius Zöllner

Abstract: Great progress has been achieved in the community of autonomous driving in the past few years. As a safety-critical problem, however, anomaly detection is a huge hurdle towards a large-scale deployment of autonomous vehicles in the real world. While many approaches, such as uncertainty estimation or segmentation-based image resynthesis, are extremely promising, there is more to be explored. Especi… ▽ More Great progress has been achieved in the community of autonomous driving in the past few years. As a safety-critical problem, however, anomaly detection is a huge hurdle towards a large-scale deployment of autonomous vehicles in the real world. While many approaches, such as uncertainty estimation or segmentation-based image resynthesis, are extremely promising, there is more to be explored. Especially inspired by works on anomaly detection based on image resynthesis, we propose a novel approach for anomaly detection through style transfer. We leverage generative models to map an image from its original style domain of road traffic to an arbitrary one and back to generate pixelwise anomaly scores. However, our experiments have proven our hypothesis wrong, and we were unable to produce significant results. Nevertheless, we want to share our findings, so that others can learn from our experiments. △ Less

Submitted 1 September, 2022; v1 submitted 13 July, 2022; originally announced July 2022.

Comments: Daniel Bogdoll and Meng Zhang contributed equally. Accepted for publication at ICECCME 2022

Knowledge Augmented Machine Learning with Applications in Autonomous Driving: A Survey

Authors: Julian Wörmann, Daniel Bogdoll, Christian Brunner, Etienne Bührle, Han Chen, Evaristus Fuh Chuo, Kostadin Cvejoski, Ludger van Elst, Philip Gottschall, Stefan Griesche, Christian Hellert, Christian Hesels, Sebastian Houben, Tim Joseph, Niklas Keil, Johann Kelsch, Mert Keser, Hendrik Königshof, Erwin Kraft, Leonie Kreuser, Kevin Krone, Tobias Latka, Denny Mattern, Stefan Matthes, Franz Motzkus , et al. (27 additional authors not shown)

Abstract: The availability of representative datasets is an essential prerequisite for many successful artificial intelligence and machine learning models. However, in real life applications these models often encounter scenarios that are inadequately represented in the data used for training. There are various reasons for the absence of sufficient data, ranging from time and cost constraints to ethical con… ▽ More The availability of representative datasets is an essential prerequisite for many successful artificial intelligence and machine learning models. However, in real life applications these models often encounter scenarios that are inadequately represented in the data used for training. There are various reasons for the absence of sufficient data, ranging from time and cost constraints to ethical considerations. As a consequence, the reliable usage of these models, especially in safety-critical applications, is still a tremendous challenge. Leveraging additional, already existing sources of knowledge is key to overcome the limitations of purely data-driven approaches. Knowledge augmented machine learning approaches offer the possibility of compensating for deficiencies, errors, or ambiguities in the data, thus increasing the generalization capability of the applied models. Even more, predictions that conform with knowledge are crucial for making trustworthy and safe decisions even in underrepresented scenarios. This work provides an overview of existing techniques and methods in the literature that combine data-driven models with existing knowledge. The identified approaches are structured according to the categories knowledge integration, extraction and conformity. In particular, we address the application of the presented methods in the field of autonomous driving. △ Less

Submitted 20 November, 2023; v1 submitted 10 May, 2022; originally announced May 2022.

Comments: 111 pages, Added section on Run-time Network Verification

Multimodal Detection of Unknown Objects on Roads for Autonomous Driving

Authors: Daniel Bogdoll, Enrico Eisen, Maximilian Nitsche, Christin Scheib, J. Marius Zöllner

Abstract: Tremendous progress in deep learning over the last years has led towards a future with autonomous vehicles on our roads. Nevertheless, the performance of their perception systems is strongly dependent on the quality of the utilized training data. As these usually only cover a fraction of all object classes an autonomous driving system will face, such systems struggle with handling the unexpected.… ▽ More Tremendous progress in deep learning over the last years has led towards a future with autonomous vehicles on our roads. Nevertheless, the performance of their perception systems is strongly dependent on the quality of the utilized training data. As these usually only cover a fraction of all object classes an autonomous driving system will face, such systems struggle with handling the unexpected. In order to safely operate on public roads, the identification of objects from unknown classes remains a crucial task. In this paper, we propose a novel pipeline to detect unknown objects. Instead of focusing on a single sensor modality, we make use of lidar and camera data by combining state-of-the art detection models in a sequential manner. We evaluate our approach on the Waymo Open Perception Dataset and point out current research gaps in anomaly detection. △ Less

Submitted 22 July, 2022; v1 submitted 3 May, 2022; originally announced May 2022.

Comments: Daniel Bogdoll, Enrico Eisen, Maximilian Nitsche, and Christin Scheib contributed equally. Accepted for publication at SMC 2022

Anomaly Detection in Autonomous Driving: A Survey

Authors: Daniel Bogdoll, Maximilian Nitsche, J. Marius Zöllner

Abstract: Nowadays, there are outstanding strides towards a future with autonomous vehicles on our roads. While the perception of autonomous vehicles performs well under closed-set conditions, they still struggle to handle the unexpected. This survey provides an extensive overview of anomaly detection techniques based on camera, lidar, radar, multimodal and abstract object level data. We provide a systemati… ▽ More Nowadays, there are outstanding strides towards a future with autonomous vehicles on our roads. While the perception of autonomous vehicles performs well under closed-set conditions, they still struggle to handle the unexpected. This survey provides an extensive overview of anomaly detection techniques based on camera, lidar, radar, multimodal and abstract object level data. We provide a systematization including detection approach, corner case level, ability for an online application, and further attributes. We outline the state-of-the-art and point out current research gaps. △ Less

Submitted 17 April, 2022; originally announced April 2022.

Comments: Daniel Bogdoll and Maximilian Nitsche contributed equally. Accepted for publication at CVPR 2022 WAD workshop

Quantification of Actual Road User Behavior on the Basis of Given Traffic Rules

Authors: Daniel Bogdoll, Moritz Nekolla, Tim Joseph, J. Marius Zöllner

Abstract: Driving on roads is restricted by various traffic rules, aiming to ensure safety for all traffic participants. However, human road users usually do not adhere to these rules strictly, resulting in varying degrees of rule conformity. Such deviations from given rules are key components of today's road traffic. In autonomous driving, robotic agents can disturb traffic flow, when rule deviations are n… ▽ More Driving on roads is restricted by various traffic rules, aiming to ensure safety for all traffic participants. However, human road users usually do not adhere to these rules strictly, resulting in varying degrees of rule conformity. Such deviations from given rules are key components of today's road traffic. In autonomous driving, robotic agents can disturb traffic flow, when rule deviations are not taken into account. In this paper, we present an approach to derive the distribution of degrees of rule conformity from human driving data. We demonstrate our method with the Waymo Open Motion dataset and Safety Distance and Speed Limit rules. △ Less

Submitted 11 May, 2022; v1 submitted 7 February, 2022; originally announced February 2022.

Comments: Daniel Bogdoll and Moritz Nekolla contributed equally. Accepted for publication at IV 2022

Ad-datasets: a meta-collection of data sets for autonomous driving

Authors: Daniel Bogdoll, Felix Schreyer, J. Marius Zöllner

Abstract: Autonomous driving is among the largest domains in which deep learning has been fundamental for progress within the last years. The rise of datasets went hand in hand with this development. All the more striking is the fact that researchers do not have a tool available that provides a quick, comprehensive and up-to-date overview of data sets and their features in the domain of autonomous driving.… ▽ More Autonomous driving is among the largest domains in which deep learning has been fundamental for progress within the last years. The rise of datasets went hand in hand with this development. All the more striking is the fact that researchers do not have a tool available that provides a quick, comprehensive and up-to-date overview of data sets and their features in the domain of autonomous driving. In this paper, we present ad-datasets, an online tool that provides such an overview for more than 150 data sets. The tool enables users to sort and filter the data sets according to currently 16 different categories. ad-datasets is an open-source project with community contributions. It is in constant development, ensuring that the content stays up-to-date. △ Less

Submitted 3 May, 2022; v1 submitted 3 February, 2022; originally announced February 2022.

Comments: Daniel Bogdoll and Felix Schreyer contributed equally. Accepted for publication at VEHITS 2022

Compressing Sensor Data for Remote Assistance of Autonomous Vehicles using Deep Generative Models

Authors: Daniel Bogdoll, Johannes Jestram, Jonas Rauch, Christin Scheib, Moritz Wittig, J. Marius Zöllner

Abstract: In the foreseeable future, autonomous vehicles will require human assistance in situations they can not resolve on their own. In such scenarios, remote assistance from a human can provide the required input for the vehicle to continue its operation. Typical sensors used in autonomous vehicles include camera and lidar sensors. Due to the massive volume of sensor data that must be sent in real-time,… ▽ More In the foreseeable future, autonomous vehicles will require human assistance in situations they can not resolve on their own. In such scenarios, remote assistance from a human can provide the required input for the vehicle to continue its operation. Typical sensors used in autonomous vehicles include camera and lidar sensors. Due to the massive volume of sensor data that must be sent in real-time, highly efficient data compression is elementary to prevent an overload of network infrastructure. Sensor data compression using deep generative neural networks has been shown to outperform traditional compression approaches for both image and lidar data, regarding compression rate as well as reconstruction quality. However, there is a lack of research about the performance of generative-neural-network-based compression algorithms for remote assistance. In order to gain insights into the feasibility of deep generative models for usage in remote assistance, we evaluate state-of-the-art algorithms regarding their applicability and identify potential weaknesses. Further, we implement an online pipeline for processing sensor data and demonstrate its performance for remote assistance using the CARLA simulator. △ Less

Submitted 9 November, 2021; v1 submitted 4 November, 2021; originally announced November 2021.

Comments: Daniel Bogdoll, Johannes Jestram, Jonas Rauch, Christin Scheib and Moritz Wittig contributed equally. Accepted for publication at NeurIPS 2021 ML4AD Workshop

Description of Corner Cases in Automated Driving: Goals and Challenges

Authors: Daniel Bogdoll, Jasmin Breitenstein, Florian Heidecker, Maarten Bieshaar, Bernhard Sick, Tim Fingscheidt, J. Marius Zöllner

Abstract: Scaling the distribution of automated vehicles requires handling various unexpected and possibly dangerous situations, termed corner cases (CC). Since many modules of automated driving systems are based on machine learning (ML), CC are an essential part of the data for their development. However, there is only a limited amount of CC data in large-scale data collections, which makes them challengin… ▽ More Scaling the distribution of automated vehicles requires handling various unexpected and possibly dangerous situations, termed corner cases (CC). Since many modules of automated driving systems are based on machine learning (ML), CC are an essential part of the data for their development. However, there is only a limited amount of CC data in large-scale data collections, which makes them challenging in the context of ML. With a better understanding of CC, offline applications, e.g., dataset analysis, and online methods, e.g., improved performance of automated driving systems, can be improved. While there are knowledge-based descriptions and taxonomies for CC, there is little research on machine-interpretable descriptions. In this extended abstract, we will give a brief overview of the challenges and goals of such a description. △ Less

Submitted 28 September, 2021; v1 submitted 20 September, 2021; originally announced September 2021.

Comments: Daniel Bogdoll, Jasmin Breitenstein and Florian Heidecker contributed equally. Accepted for publication at ICCV 2021 ERCVAD Workshop

Taxonomy and Survey on Remote Human Input Systems for Driving Automation Systems

Authors: Daniel Bogdoll, Stefan Orf, Lars Töttel, J. Marius Zöllner

Abstract: Corner cases for driving automation systems can often be detected by the system itself and subsequently resolved by remote humans. There exists a wide variety of technical approaches on how remote humans can resolve such issues. Over multiple domains, no common taxonomy on those approaches has developed yet, though. As the scaling of automated driving systems continues to increase, a uniform taxon… ▽ More Corner cases for driving automation systems can often be detected by the system itself and subsequently resolved by remote humans. There exists a wide variety of technical approaches on how remote humans can resolve such issues. Over multiple domains, no common taxonomy on those approaches has developed yet, though. As the scaling of automated driving systems continues to increase, a uniform taxonomy is desirable to improve communication within the scientific community, but also beyond to policymakers and the general public. In this paper, we provide a survey on recent terminologies and propose a taxonomy for remote human input systems, classifying the different approaches based on their complexity. △ Less

Submitted 10 March, 2022; v1 submitted 17 September, 2021; originally announced September 2021.

Comments: Accepted for publication at FICC 2022

KIGLIS: Smart Networks for Smart Cities

Authors: Daniel Bogdoll, Patrick Matalla, Christoph Füllner, Christian Raack, Shi Li, Tobias Käfer, Stefan Orf, Marc René Zofka, Finn Sartoris, Christoph Schweikert, Thomas Pfeiffer, André Richter, Sebastian Randel, Rene Bonk

Abstract: Smart cities will be characterized by a variety of intelligent and networked services, each with specific requirements for the underlying network infrastructure. While smart city architectures and services have been studied extensively, little attention has been paid to the network technology. The KIGLIS research project, consisting of a consortium of companies, universities and research instituti… ▽ More Smart cities will be characterized by a variety of intelligent and networked services, each with specific requirements for the underlying network infrastructure. While smart city architectures and services have been studied extensively, little attention has been paid to the network technology. The KIGLIS research project, consisting of a consortium of companies, universities and research institutions, focuses on artificial intelligence for optimizing fiber-optic networks of a smart city, with a special focus on future mobility applications, such as automated driving. In this paper, we present early results on our process of collecting smart city requirements for communication networks, which will lead towards reference infrastructure and architecture solutions. Finally, we suggest directions in which artificial intelligence will improve smart city networks. △ Less

Submitted 28 September, 2021; v1 submitted 14 May, 2021; originally announced June 2021.

Comments: Accepted for publication at ISC2 2021

Towards Sensor Data Abstraction of Autonomous Vehicle Perception Systems

Authors: Hannes Reichert, Lukas Lang, Kevin Rösch, Daniel Bogdoll, Konrad Doll, Bernhard Sick, Hans-Christian Reuss, Christoph Stiller, J. Marius Zöllner

Abstract: Full-stack autonomous driving perception modules usually consist of data-driven models based on multiple sensor modalities. However, these models might be biased to the sensor setup used for data acquisition. This bias can seriously impair the perception models' transferability to new sensor setups, which continuously occur due to the market's competitive nature. We envision sensor data abstractio… ▽ More Full-stack autonomous driving perception modules usually consist of data-driven models based on multiple sensor modalities. However, these models might be biased to the sensor setup used for data acquisition. This bias can seriously impair the perception models' transferability to new sensor setups, which continuously occur due to the market's competitive nature. We envision sensor data abstraction as an interface between sensor data and machine learning applications for highly automated vehicles (HAD). For this purpose, we review the primary sensor modalities, camera, lidar, and radar, published in autonomous-driving related datasets, examine single sensor abstraction and abstraction of sensor setups, and identify critical paths towards an abstraction of sensor data from multiple perception configurations. △ Less

Submitted 28 September, 2021; v1 submitted 14 May, 2021; originally announced May 2021.

Comments: Hannes Reichert, Lukas Lang, Kevin Rösch and Daniel Bogdoll contributed equally. Accepted for publication at ISC2 2021

Reliving the Dataset: Combining the Visualization of Road Users' Interactions with Scenario Reconstruction in Virtual Reality

Authors: Lars Töttel, Maximilian Zipfl, Daniel Bogdoll, Marc René Zofka, J. Marius Zöllner

Abstract: One core challenge in the development of automated vehicles is their capability to deal with a multitude of complex trafficscenarios with many, hard to predict traffic participants. As part of the iterative development process, it is necessary to detect criticalscenarios and generate knowledge from them to improve the highly automated driving (HAD) function. In order to tackle this challenge,numer… ▽ More One core challenge in the development of automated vehicles is their capability to deal with a multitude of complex trafficscenarios with many, hard to predict traffic participants. As part of the iterative development process, it is necessary to detect criticalscenarios and generate knowledge from them to improve the highly automated driving (HAD) function. In order to tackle this challenge,numerous datasets have been released in the past years, which act as the basis for the development and testing of such algorithms.Nevertheless, the remaining challenges are to find relevant scenes, such as safety-critical corner cases, in these datasets and tounderstand them completely.Therefore, this paper presents a methodology to process and analyze naturalistic motion datasets in two ways: On the one hand, ourapproach maps scenes of the datasets to a generic semantic scene graph which allows for a high-level and objective analysis. Here,arbitrary criticality measures, e.g. TTC, RSS or SFF, can be set to automatically detect critical scenarios between traffic participants.On the other hand, the scenarios are recreated in a realistic virtual reality (VR) environment, which allows for a subjective close-upanalysis from multiple, interactive perspectives. △ Less

Submitted 30 September, 2021; v1 submitted 4 May, 2021; originally announced May 2021.

Comments: Accepted for publication at ICITE 2021