Showing 1–45 of 45 results for author: Buchholz, M

Generalized Coordination of Partially Cooperative Urban Traffic

Authors: Max Bastian Mertens, Michael Buchholz

Abstract: Vehicle-to-anything connectivity, especially for autonomous vehicles, promises to increase passenger comfort and safety of road traffic, for example, by sharing perception and driving intention. Cooperative maneuver planning uses connectivity to enhance traffic efficiency, which has, so far, been mainly considered for automated intersection management. In this article, we present a novel cooperati… ▽ More Vehicle-to-anything connectivity, especially for autonomous vehicles, promises to increase passenger comfort and safety of road traffic, for example, by sharing perception and driving intention. Cooperative maneuver planning uses connectivity to enhance traffic efficiency, which has, so far, been mainly considered for automated intersection management. In this article, we present a novel cooperative maneuver planning approach that is generalized to various situations found in urban traffic. Our framework handles challenging mixed traffic, that is, traffic comprising both cooperative connected vehicles and other vehicles at any distribution. Our solution is based on an optimization approach accompanied by an efficient heuristic method for high-load scenarios. We extensively evaluate the proposed planer in a distinctly realistic simulation framework and show significant efficiency gains already at a cooperation rate of 40%. Traffic throughput increases, while the average waiting time and the number of stopped vehicles are reduced, without impacting traffic safety. △ Less

Submitted 27 May, 2025; originally announced May 2025.

Comments: 8 pages, 7 figures, 3 tables, submitted to IEEE Transactions on Intelligent Vehicles (T-IV)

AI Safety Assurance for Automated Vehicles: A Survey on Research, Standardization, Regulation

Authors: Lars Ullrich, Michael Buchholz, Klaus Dietmayer, Knut Graichen

Abstract: Assuring safety of artificial intelligence (AI) applied to safety-critical systems is of paramount importance. Especially since research in the field of automated driving shows that AI is able to outperform classical approaches, to handle higher complexities, and to reach new levels of autonomy. At the same time, the safety assurance required for the use of AI in such safety-critical systems is st… ▽ More Assuring safety of artificial intelligence (AI) applied to safety-critical systems is of paramount importance. Especially since research in the field of automated driving shows that AI is able to outperform classical approaches, to handle higher complexities, and to reach new levels of autonomy. At the same time, the safety assurance required for the use of AI in such safety-critical systems is still not in place. Due to the dynamic and far-reaching nature of the technology, research on safeguarding AI is being conducted in parallel to AI standardization and regulation. The parallel progress necessitates simultaneous consideration in order to carry out targeted research and development of AI systems in the context of automated driving. Therefore, in contrast to existing surveys that focus primarily on research aspects, this paper considers research, standardization and regulation in a concise way. Accordingly, the survey takes into account the interdependencies arising from the triplet of research, standardization and regulation in a forward-looking perspective and anticipates and discusses open questions and possible future directions. In this way, the survey ultimately serves to provide researchers and safety experts with a compact, holistic perspective that discusses the current status, emerging trends, and possible future developments. △ Less

Submitted 25 April, 2025; originally announced April 2025.

Comments: Published in IEEE Transactions on Intelligent Vehicles,15 November 2024

Journal ref: IEEE Transactions on Intelligent Vehicles 2024

Dynamic Objective MPC for Motion Planning of Seamless Docking Maneuvers

Authors: Oliver Schumann, Michael Buchholz, Klaus Dietmayer

Abstract: Automated vehicles and logistics robots must often position themselves in narrow environments with high precision in front of a specific target, such as a package or their charging station. Often, these docking scenarios are solved in two steps: path following and rough positioning followed by a high-precision motion planning algorithm. This can generate suboptimal trajectories caused by bad posit… ▽ More Automated vehicles and logistics robots must often position themselves in narrow environments with high precision in front of a specific target, such as a package or their charging station. Often, these docking scenarios are solved in two steps: path following and rough positioning followed by a high-precision motion planning algorithm. This can generate suboptimal trajectories caused by bad positioning in the first phase and, therefore, prolong the time it takes to reach the goal. In this work, we propose a unified approach, which is based on a Model Predictive Control (MPC) that unifies the advantages of Model Predictive Contouring Control (MPCC) with a Cartesian MPC to reach a specific goal pose. The paper's main contributions are the adaption of the dynamic weight allocation method to reach path ends and goal poses inside driving corridors, and the development of the so-called dynamic objective MPC. The latter is an improvement of the dynamic weight allocation method, which can inherently switch state-dependent from an MPCC to a Cartesian MPC to solve the path-following problem and the high-precision positioning tasks independently of the location of the goal pose seamlessly by one algorithm. This leads to foresighted, feasible, and safe motion plans, which can decrease the mission time and result in smoother trajectories. △ Less

Submitted 4 April, 2025; originally announced April 2025.

Comments: accepted for publication at 2025 IEEE Intelligent Vehicles Symposium (IV)

Expanding the Classical V-Model for the Development of Complex Systems Incorporating AI

Authors: Lars Ullrich, Michael Buchholz, Klaus Dietmayer, Knut Graichen

Abstract: Research in the field of automated vehicles, or more generally cognitive cyber-physical systems that operate in the real world, is leading to increasingly complex systems. Among other things, artificial intelligence enables an ever-increasing degree of autonomy. In this context, the V-model, which has served for decades as a process reference model of the system development lifecycle is reaching i… ▽ More Research in the field of automated vehicles, or more generally cognitive cyber-physical systems that operate in the real world, is leading to increasingly complex systems. Among other things, artificial intelligence enables an ever-increasing degree of autonomy. In this context, the V-model, which has served for decades as a process reference model of the system development lifecycle is reaching its limits. To the contrary, innovative processes and frameworks have been developed that take into account the characteristics of emerging autonomous systems. To bridge the gap and merge the different methodologies, we present an extension of the V-model for iterative data-based development processes that harmonizes and formalizes the existing methods towards a generic framework. The iterative approach allows for seamless integration of continuous system refinement. While the data-based approach constitutes the consideration of data-based development processes and formalizes the use of synthetic and real world data. In this way, formalizing the process of development, verification, validation, and continuous integration contributes to ensuring the safety of emerging complex systems that incorporate AI. △ Less

Submitted 18 February, 2025; originally announced February 2025.

Comments: Published in IEEE Transactions on Intelligent Vehicles, 29 July 2024

Journal ref: IEEE Transactions on Intelligent Vehicles 2024

MGNiceNet: Unified Monocular Geometric Scene Understanding

Authors: Markus Schön, Michael Buchholz, Klaus Dietmayer

Abstract: Monocular geometric scene understanding combines panoptic segmentation and self-supervised depth estimation, focusing on real-time application in autonomous vehicles. We introduce MGNiceNet, a unified approach that uses a linked kernel formulation for panoptic segmentation and self-supervised depth estimation. MGNiceNet is based on the state-of-the-art real-time panoptic segmentation method RT-K-N… ▽ More Monocular geometric scene understanding combines panoptic segmentation and self-supervised depth estimation, focusing on real-time application in autonomous vehicles. We introduce MGNiceNet, a unified approach that uses a linked kernel formulation for panoptic segmentation and self-supervised depth estimation. MGNiceNet is based on the state-of-the-art real-time panoptic segmentation method RT-K-Net and extends the architecture to cover both panoptic segmentation and self-supervised monocular depth estimation. To this end, we introduce a tightly coupled self-supervised depth estimation predictor that explicitly uses information from the panoptic path for depth prediction. Furthermore, we introduce a panoptic-guided motion masking method to improve depth estimation without relying on video panoptic segmentation annotations. We evaluate our method on two popular autonomous driving datasets, Cityscapes and KITTI. Our model shows state-of-the-art results compared to other real-time methods and closes the gap to computationally more demanding methods. Source code and trained models are available at https://github.com/markusschoen/MGNiceNet. △ Less

Submitted 10 March, 2025; v1 submitted 18 November, 2024; originally announced November 2024.

Journal ref: Proceedings of the Asian Conference on Computer Vision (ACCV), 2024, pp. 1502-1519

The ADUULM-360 Dataset -- A Multi-Modal Dataset for Depth Estimation in Adverse Weather

Authors: Markus Schön, Jona Ruof, Thomas Wodtko, Michael Buchholz, Klaus Dietmayer

Abstract: Depth estimation is an essential task toward full scene understanding since it allows the projection of rich semantic information captured by cameras into 3D space. While the field has gained much attention recently, datasets for depth estimation lack scene diversity or sensor modalities. This work presents the ADUULM-360 dataset, a novel multi-modal dataset for depth estimation. The ADUULM-360 da… ▽ More Depth estimation is an essential task toward full scene understanding since it allows the projection of rich semantic information captured by cameras into 3D space. While the field has gained much attention recently, datasets for depth estimation lack scene diversity or sensor modalities. This work presents the ADUULM-360 dataset, a novel multi-modal dataset for depth estimation. The ADUULM-360 dataset covers all established autonomous driving sensor modalities, cameras, lidars, and radars. It covers a frontal-facing stereo setup, six surround cameras covering the full 360-degree, two high-resolution long-range lidar sensors, and five long-range radar sensors. It is also the first depth estimation dataset that contains diverse scenes in good and adverse weather conditions. We conduct extensive experiments using state-of-the-art self-supervised depth estimation methods under different training tasks, such as monocular training, stereo training, and full surround training. Discussing these results, we demonstrate common limitations of state-of-the-art methods, especially in adverse weather conditions, which hopefully will inspire future research in this area. Our dataset, development kit, and trained baselines are available at https://github.com/uulm-mrm/aduulm_360_dataset. △ Less

Submitted 18 November, 2024; originally announced November 2024.

Comments: 2024 IEEE International Conference on Intelligent Transportation Systems (ITSC)

Self-Assessment of Evidential Grid Map Fusion for Robust Motion Planning

Authors: Oliver Schumann, Thomas Wodtko, Michael Buchholz, Klaus Dietmayer

Abstract: Conflicting sensor measurements pose a huge problem for the environment representation of an autonomous robot. Therefore, in this paper, we address the self-assessment of an evidential grid map in which data from conflicting LiDAR sensor measurements are fused, followed by methods for robust motion planning under these circumstances. First, conflicting measurements aggregated in Subjective-Logic-b… ▽ More Conflicting sensor measurements pose a huge problem for the environment representation of an autonomous robot. Therefore, in this paper, we address the self-assessment of an evidential grid map in which data from conflicting LiDAR sensor measurements are fused, followed by methods for robust motion planning under these circumstances. First, conflicting measurements aggregated in Subjective-Logic-based evidential grid maps are classified. Then, a self-assessment framework evaluates these conflicts and estimates their severity for the overall system by calculating a degradation score. This enables the detection of calibration errors and insufficient sensor setups. In contrast to other motion planning approaches, the information gained from the evidential grid maps is further used inside our proposed path-planning algorithm. Here, the impact of conflicting measurements on the current motion plan is evaluated, and a robust and curious path-planning strategy is derived to plan paths under the influence of conflicting data. This ensures that the system integrity is maintained in severely degraded environment representations which can prevent the unnecessary abortion of planning tasks. △ Less

Submitted 30 September, 2024; originally announced September 2024.

Comments: Oliver Schumann, Thomas Wodtko, Michael Buchholz, Klaus Dietmayer

Self-Assessment and Correction of Sensor Synchronization

Authors: Thomas Wodtko, Alexander Scheible, Michael Buchholz

Abstract: We propose an approach to assess the synchronization of rigidly mounted sensors based on their rotational motion. Using function similarity measures combined with a sliding window approach, our approach is capable of estimating time-varying time offsets. Further, the estimated offset allows the correction of erroneously assigned time stamps on measurements. This mitigates the effect of synchroniza… ▽ More We propose an approach to assess the synchronization of rigidly mounted sensors based on their rotational motion. Using function similarity measures combined with a sliding window approach, our approach is capable of estimating time-varying time offsets. Further, the estimated offset allows the correction of erroneously assigned time stamps on measurements. This mitigates the effect of synchronization issues on subsequent modules in autonomous software stacks, such as tracking systems that heavily rely on accurate measurement time stamps. Additionally, a self-assessment based on an uncertainty measure is derived, and correction strategies are described. Our approach is evaluated with Monte Carlo experiments containing different error patterns. The results show that our approach accurately estimates time offsets and, thus, is able to detect and assess synchronization issues. To further embrace the importance of our approach for autonomous systems, we investigate the effect of synchronization inconsistencies in tracking systems in more detail and demonstrate the beneficial effect of our proposed offset correction. △ Less

Submitted 30 September, 2024; originally announced September 2024.

Globally Optimal GNSS Multi-Antenna Lever Arm Calibration

Authors: Thomas Wodtko, Michael Buchholz

Abstract: Sensor calibration is crucial for autonomous driving, providing the basis for accurate localization and consistent data fusion. Enabling the use of high-accuracy GNSS sensors, this work focuses on the antenna lever arm calibration. We propose a globally optimal multi-antenna lever arm calibration approach based on motion measurements. For this, we derive an optimization method that further allows… ▽ More Sensor calibration is crucial for autonomous driving, providing the basis for accurate localization and consistent data fusion. Enabling the use of high-accuracy GNSS sensors, this work focuses on the antenna lever arm calibration. We propose a globally optimal multi-antenna lever arm calibration approach based on motion measurements. For this, we derive an optimization method that further allows the integration of a-priori knowledge. Globally optimal solutions are obtained by leveraging the Lagrangian dual problem and a primal recovery strategy. Generally, motion-based calibration for autonomous vehicles is known to be difficult due to cars' predominantly planar motion. Therefore, we first describe the motion requirements for a unique solution and then propose a planar motion extension to overcome this issue and enable a calibration based on the restricted motion of autonomous vehicles. Last we present and discuss the results of our thorough evaluation. Using simulated and augmented real-world data, we achieve accurate calibration results and fast run times that allow online deployment. △ Less

Submitted 30 September, 2024; v1 submitted 14 June, 2024; originally announced June 2024.

Real-World Evaluation of two Cooperative Intersection Management Approaches

Authors: Marvin Klimke, Max Bastian Mertens, Benjamin Völz, Michael Buchholz

Abstract: Cooperative maneuver planning promises to significantly improve traffic efficiency at unsignalized intersections by leveraging connected automated vehicles. Previous works on this topic have been mostly developed for completely automated traffic in a simple simulated environment. In contrast, our previously introduced planning approaches are specifically designed to handle real-world mixed traffic… ▽ More Cooperative maneuver planning promises to significantly improve traffic efficiency at unsignalized intersections by leveraging connected automated vehicles. Previous works on this topic have been mostly developed for completely automated traffic in a simple simulated environment. In contrast, our previously introduced planning approaches are specifically designed to handle real-world mixed traffic. The two methods are based on multi-scenario prediction and graph-based reinforcement learning, respectively. This is the first study to perform evaluations in a novel mixed traffic simulation framework as well as real-world drives with prototype connected automated vehicles in public traffic. The simulation features the same connected automated driving software stack as deployed on one of the automated vehicles. Our quantitative evaluations show that cooperative maneuver planning achieves a substantial reduction in crossing times and the number of stops. In a realistic environment with few automated vehicles, there are noticeable efficiency gains with only slightly increasing criticality metrics. △ Less

Submitted 14 April, 2025; v1 submitted 25 March, 2024; originally announced March 2024.

Comments: M. Klimke and M. B. Mertens are both first authors with equal contribution. 10 pages, 9 figures, 3 tables, submitted to IEEE Intelligent Transportation Systems Magazine

Fast Long-Term Multi-Scenario Prediction for Maneuver Planning at Unsignalized Intersections

Authors: Max Bastian Mertens, Jona Ruof, Jan Strohbeck, Michael Buchholz

Abstract: Motion prediction for intelligent vehicles typically focuses on estimating the most probable future evolutions of a traffic scenario. Estimating the gap acceptance, i.e., whether a vehicle merges or crosses before another vehicle with the right of way, is often handled implicitly in the prediction. However, an infrastructure-based maneuver planning can assign artificial priorities between cooperat… ▽ More Motion prediction for intelligent vehicles typically focuses on estimating the most probable future evolutions of a traffic scenario. Estimating the gap acceptance, i.e., whether a vehicle merges or crosses before another vehicle with the right of way, is often handled implicitly in the prediction. However, an infrastructure-based maneuver planning can assign artificial priorities between cooperative vehicles, so it needs to evaluate many more potential scenarios. Additionally, the prediction horizon has to be long enough to assess the impact of a maneuver. We, therefore, present a novel long-term prediction approach handling the gap acceptance estimation and the velocity prediction in two separate stages. Thereby, the behavior of regular vehicles as well as priority assignments of cooperative vehicles can be considered. We train both stages on real-world traffic observations to achieve realistic prediction results. Our method has a competitive accuracy and is fast enough to predict a multitude of scenarios in a short time, making it suitable to be used in a maneuver planning framework. △ Less

Submitted 17 September, 2024; v1 submitted 26 January, 2024; originally announced January 2024.

Journal ref: 2024 American Control Conference (ACC), Publication Year: 2024, Pages: 111-116

Graph-based Trajectory Prediction with Cooperative Information

Authors: Jan Strohbeck, Sebastian Maschke, Max Mertens, Michael Buchholz

Abstract: For automated driving, predicting the future trajectories of other road users in complex traffic situations is a hard problem. Modern neural networks use the past trajectories of traffic participants as well as map data to gather hints about the possible driver intention and likely maneuvers. With increasing connectivity between cars and other traffic actors, cooperative information is another sou… ▽ More For automated driving, predicting the future trajectories of other road users in complex traffic situations is a hard problem. Modern neural networks use the past trajectories of traffic participants as well as map data to gather hints about the possible driver intention and likely maneuvers. With increasing connectivity between cars and other traffic actors, cooperative information is another source of data that can be used as inputs for trajectory prediction algorithms. Connected actors might transmit their intended path or even complete planned trajectories to other actors, which simplifies the prediction problem due to the imposed constraints. In this work, we outline the benefits of using this source of data for trajectory prediction and propose a graph-based neural network architecture that can leverage this additional data. We show that the network performance increases substantially if cooperative data is present. Also, our proposed training scheme improves the network's performance even for cases where no cooperative information is available. We also show that the network can deal with inaccurate cooperative data, which allows it to be used in real automated driving environments. △ Less

Submitted 24 October, 2023; originally announced October 2023.

Comments: Accepted for publication at the 26th IEEE International Conference on Intelligent Transportation Systems 2023

Efficient Path Planning in Large Unknown Environments with Switchable System Models for Automated Vehicles

Authors: Oliver Schumann, Michael Buchholz, Klaus Dietmayer

Abstract: Large environments are challenging for path planning algorithms as the size of the configuration space increases. Furthermore, if the environment is mainly unexplored, large amounts of the path are planned through unknown areas. Hence, a complete replanning of the entire path occurs whenever the path collides with newly discovered obstacles. We propose a novel method that stops the path planning a… ▽ More Large environments are challenging for path planning algorithms as the size of the configuration space increases. Furthermore, if the environment is mainly unexplored, large amounts of the path are planned through unknown areas. Hence, a complete replanning of the entire path occurs whenever the path collides with newly discovered obstacles. We propose a novel method that stops the path planning algorithm after a certain distance. It is used to navigate the algorithm in large environments and is not prone to problems of existing navigation approaches. Furthermore, we developed a method to detect significant environment changes to allow a more efficient replanning. At last, we extend the path planner to be used in the U-Shift concept vehicle. It can switch to another system model and rotate around the center of its rear axis. The results show that the proposed methods generate nearly identical paths compared to the standard Hybrid A* while drastically reducing the execution time. Furthermore, we show that the extended path planning algorithm enables the efficient use of the maneuvering capabilities of the concept vehicle to plan concise paths in narrow environments. △ Less

Submitted 30 September, 2024; v1 submitted 10 October, 2023; originally announced October 2023.

User Feedback and Sample Weighting for Ill-Conditioned Hand-Eye Calibration

Authors: Markus Horn, Thomas Wodtko, Michael Buchholz, Klaus Dietmayer

Abstract: Hand-eye calibration is an important and extensively researched method for calibrating rigidly coupled sensors, solely based on estimates of their motion. Due to the geometric structure of this problem, at least two motion estimates with non-parallel rotation axes are required for a unique solution. If the majority of rotation axes are almost parallel, the resulting optimization problem is ill-con… ▽ More Hand-eye calibration is an important and extensively researched method for calibrating rigidly coupled sensors, solely based on estimates of their motion. Due to the geometric structure of this problem, at least two motion estimates with non-parallel rotation axes are required for a unique solution. If the majority of rotation axes are almost parallel, the resulting optimization problem is ill-conditioned. In this paper, we propose an approach to automatically weight the motion samples of such an ill-conditioned optimization problem for improving the conditioning. The sample weights are chosen in relation to the local density of all available rotation axes. Furthermore, we present an approach for estimating the sensitivity and conditioning of the cost function, separated into the translation and the rotation part. This information can be employed as user feedback when recording the calibration data to prevent ill-conditioning in advance. We evaluate and compare our approach on artificially augmented data from the KITTI odometry dataset. △ Less

Submitted 11 August, 2023; originally announced August 2023.

Comments: Accepted for publication at the 26th IEEE International Conference on Intelligent Transportation Systems 2023

Adaptive Patched Grid Mapping

Authors: Thomas Wodtko, Thomas Griebel, Michael Buchholz

Abstract: In this work, we propose a novel adaptive grid mapping approach, the Adaptive Patched Grid Map, which enables a situational aware grid based perception for autonomous vehicles. Its structure allows a flexible representation of the surrounding unstructured environment. By splitting types of information into separate layers less memory is allocated when data is unevenly or sporadically available. Ho… ▽ More In this work, we propose a novel adaptive grid mapping approach, the Adaptive Patched Grid Map, which enables a situational aware grid based perception for autonomous vehicles. Its structure allows a flexible representation of the surrounding unstructured environment. By splitting types of information into separate layers less memory is allocated when data is unevenly or sporadically available. However, layers must be resampled during the fusion process to cope with dynamically changing cell sizes. Therefore, we propose a novel spatial cell fusion approach. Together with the proposed fusion framework, dynamically changing external requirements, such as cell resolution specifications and horizon targets, are considered. For our evaluation, real-world data were recorded from an autonomous vehicle driving through various traffic situations. Based on this, the memory efficiency is compared to other approaches, and fusion execution times are determined. The results confirm the adaptation to requirement changes and a significant memory usage reduction. △ Less

Submitted 7 August, 2023; originally announced August 2023.

Advancing Frame-Dropping in Multi-Object Tracking-by-Detection Systems Through Event-Based Detection Triggering

Authors: Matti Henning, Michael Buchholz, Klaus Dietmayer

Abstract: With rising computational requirements modern automated vehicles (AVs) often consider trade-offs between energy consumption and perception performance, potentially jeopardizing their safe operation. Frame-dropping in tracking-by-detection perception systems presents a promising approach, although late traffic participant detection might be induced. In this paper, we extend our previous work on f… ▽ More With rising computational requirements modern automated vehicles (AVs) often consider trade-offs between energy consumption and perception performance, potentially jeopardizing their safe operation. Frame-dropping in tracking-by-detection perception systems presents a promising approach, although late traffic participant detection might be induced. In this paper, we extend our previous work on frame-dropping in tracking-by-detection perception systems. We introduce an additional event-based triggering mechanism using camera object detections to increase both the system's efficiency, as well as its safety. Evaluating both single and multi-modal tracking methods we show that late object detections are mitigated while the potential for reduced energy consumption is significantly increased, reaching nearly 60 Watt per reduced point in HOTA score. △ Less

Submitted 1 August, 2023; originally announced August 2023.

Comments: Accepted for publication at the 26th IEEE International Conference on Intelligent Transportation Systems 2023

Automated Automotive Radar Calibration With Intelligent Vehicles

Authors: Alexander Tsaregorodtsev, Michael Buchholz, Vasileios Belagiannis

Abstract: While automotive radar sensors are widely adopted and have been used for automatic cruise control and collision avoidance tasks, their application outside of vehicles is still limited. As they have the ability to resolve multiple targets in 3D space, radars can also be used for improving environment perception. This application, however, requires a precise calibration, which is usually a time-cons… ▽ More While automotive radar sensors are widely adopted and have been used for automatic cruise control and collision avoidance tasks, their application outside of vehicles is still limited. As they have the ability to resolve multiple targets in 3D space, radars can also be used for improving environment perception. This application, however, requires a precise calibration, which is usually a time-consuming and labor-intensive task. We, therefore, present an approach for automated and geo-referenced extrinsic calibration of automotive radar sensors that is based on a novel hypothesis filtering scheme. Our method does not require external modifications of a vehicle and instead uses the location data obtained from automated vehicles. This location data is then combined with filtered sensor data to create calibration hypotheses. Subsequent filtering and optimization recovers the correct calibration. Our evaluation on data from a real testing site shows that our method can correctly calibrate infrastructure sensors in an automated manner, thus enabling cooperative driving scenarios. △ Less

Submitted 23 June, 2023; originally announced June 2023.

Comments: 5 pages, 4 figures, accepted for presentation at the 31st European Signal Processing Conference (EUSIPCO), September 4 - September 8, 2023, Helsinki, Finland

Assessing the Effectiveness of GPT-3 in Detecting False Political Statements: A Case Study on the LIAR Dataset

Authors: Mars Gokturk Buchholz

Abstract: The detection of political fake statements is crucial for maintaining information integrity and preventing the spread of misinformation in society. Historically, state-of-the-art machine learning models employed various methods for detecting deceptive statements. These methods include the use of metadata (W. Wang et al., 2018), n-grams analysis (Singh et al., 2021), and linguistic (Wu et al., 2022… ▽ More The detection of political fake statements is crucial for maintaining information integrity and preventing the spread of misinformation in society. Historically, state-of-the-art machine learning models employed various methods for detecting deceptive statements. These methods include the use of metadata (W. Wang et al., 2018), n-grams analysis (Singh et al., 2021), and linguistic (Wu et al., 2022) and stylometric (Islam et al., 2020) features. Recent advancements in large language models, such as GPT-3 (Brown et al., 2020) have achieved state-of-the-art performance on a wide range of tasks. In this study, we conducted experiments with GPT-3 on the LIAR dataset (W. Wang et al., 2018) and achieved higher accuracy than state-of-the-art models without using any additional meta or linguistic features. Additionally, we experimented with zero-shot learning using a carefully designed prompt and achieved near state-of-the-art performance. An advantage of this approach is that the model provided evidence for its decision, which adds transparency to the model's decision-making and offers a chance for users to verify the validity of the evidence provided. △ Less

Submitted 13 June, 2023; originally announced June 2023.

Comments: 7 pages, 1 figure

ACM Class: I.2.7

Real-Time Spatial Trajectory Planning for Urban Environments Using Dynamic Optimization

Authors: Jona Ruof, Max Bastian Mertens, Michael Buchholz, Klaus Dietmayer

Abstract: Planning trajectories for automated vehicles in urban environments requires methods with high generality, long planning horizons, and fast update rates. Using a path-velocity decomposition, we contribute a novel planning framework, which generates foresighted trajectories and can handle a wide variety of state and control constraints effectively. In contrast to related work, the proposed optimal c… ▽ More Planning trajectories for automated vehicles in urban environments requires methods with high generality, long planning horizons, and fast update rates. Using a path-velocity decomposition, we contribute a novel planning framework, which generates foresighted trajectories and can handle a wide variety of state and control constraints effectively. In contrast to related work, the proposed optimal control problems are formulated over space rather than time. This spatial formulation decouples environmental constraints from the optimization variables, which allows the application of simple, yet efficient shooting methods. To this end, we present a tailored solution strategy based on ILQR, in the Augmented Lagrangian framework, to rapidly minimize the trajectory objective costs, even under infeasible initial solutions. Evaluations in simulation and on a full-sized automated vehicle in real-world urban traffic show the real-time capability and versatility of the proposed approach. △ Less

Submitted 9 August, 2023; v1 submitted 4 May, 2023; originally announced May 2023.

Journal ref: 2023 IEEE Intelligent Vehicles Symposium (IV)

Extrinsic Infrastructure Calibration Using the Hand-Eye Robot-World Formulation

Authors: Markus Horn, Thomas Wodtko, Michael Buchholz, Klaus Dietmayer

Abstract: We propose a certifiably globally optimal approach for solving the hand-eye robot-world problem supporting multiple sensors and targets at once. Further, we leverage this formulation for estimating a geo-referenced calibration of infrastructure sensors. Since vehicle motion recorded by infrastructure sensors is mostly planar, obtaining a unique solution for the respective hand-eye robot-world prob… ▽ More We propose a certifiably globally optimal approach for solving the hand-eye robot-world problem supporting multiple sensors and targets at once. Further, we leverage this formulation for estimating a geo-referenced calibration of infrastructure sensors. Since vehicle motion recorded by infrastructure sensors is mostly planar, obtaining a unique solution for the respective hand-eye robot-world problem is unfeasible without incorporating additional knowledge. Hence, we extend our proposed method to include a-priori knowledge, i.e., the translation norm of calibration targets, to yield a unique solution. Our approach achieves state-of-the-art results on simulated and real-world data. Especially on real-world intersection data, our approach utilizing the translation norm is the only method providing accurate results. △ Less

Submitted 2 August, 2023; v1 submitted 2 May, 2023; originally announced May 2023.

Journal ref: 2023 IEEE Intelligent Vehicles Symposium (IV)

RT-K-Net: Revisiting K-Net for Real-Time Panoptic Segmentation

Authors: Markus Schön, Michael Buchholz, Klaus Dietmayer

Abstract: Panoptic segmentation is one of the most challenging scene parsing tasks, combining the tasks of semantic segmentation and instance segmentation. While much progress has been made, few works focus on the real-time application of panoptic segmentation methods. In this paper, we revisit the recently introduced K-Net architecture. We propose vital changes to the architecture, training, and inference… ▽ More Panoptic segmentation is one of the most challenging scene parsing tasks, combining the tasks of semantic segmentation and instance segmentation. While much progress has been made, few works focus on the real-time application of panoptic segmentation methods. In this paper, we revisit the recently introduced K-Net architecture. We propose vital changes to the architecture, training, and inference procedure, which massively decrease latency and improve performance. Our resulting RT-K-Net sets a new state-of-the-art performance for real-time panoptic segmentation methods on the Cityscapes dataset and shows promising results on the challenging Mapillary Vistas dataset. On Cityscapes, RT-K-Net reaches 60.2 % PQ with an average inference time of 32 ms for full resolution 1024x2048 pixel images on a single Titan RTX GPU. On Mapillary Vistas, RT-K-Net reaches 33.2 % PQ with an average inference time of 69 ms. Source code is available at https://github.com/markusschoen/RT-K-Net. △ Less

Submitted 4 August, 2023; v1 submitted 2 May, 2023; originally announced May 2023.

Journal ref: 2023 IEEE Intelligent Vehicles Symposium (IV)

Automated Static Camera Calibration with Intelligent Vehicles

Authors: Alexander Tsaregorodtsev, Adrian Holzbock, Jan Strohbeck, Michael Buchholz, Vasileios Belagiannis

Abstract: Connected and cooperative driving requires precise calibration of the roadside infrastructure for having a reliable perception system. To solve this requirement in an automated manner, we present a robust extrinsic calibration method for automated geo-referenced camera calibration. Our method requires a calibration vehicle equipped with a combined GNSS/RTK receiver and an inertial measurement unit… ▽ More Connected and cooperative driving requires precise calibration of the roadside infrastructure for having a reliable perception system. To solve this requirement in an automated manner, we present a robust extrinsic calibration method for automated geo-referenced camera calibration. Our method requires a calibration vehicle equipped with a combined GNSS/RTK receiver and an inertial measurement unit (IMU) for self-localization. In order to remove any requirements for the target's appearance and the local traffic conditions, we propose a novel approach using hypothesis filtering. Our method does not require any human interaction with the information recorded by both the infrastructure and the vehicle. Furthermore, we do not limit road access for other road users during calibration. We demonstrate the feasibility and accuracy of our approach by evaluating our approach on synthetic datasets as well as a real-world connected intersection, and deploying the calibration on real infrastructure. Our source code is publicly available. △ Less

Submitted 21 April, 2023; originally announced April 2023.

Comments: 7 pages, 3 figures, accepted for presentation at the 34th IEEE Intelligent Vehicles Symposium (IV 2023), June 4 - June 7, 2023, Anchorage, Alaska, United States of America

Integration of Reinforcement Learning Based Behavior Planning With Sampling Based Motion Planning for Automated Driving

Authors: Marvin Klimke, Benjamin Völz, Michael Buchholz

Abstract: Reinforcement learning has received high research interest for developing planning approaches in automated driving. Most prior works consider the end-to-end planning task that yields direct control commands and rarely deploy their algorithm to real vehicles. In this work, we propose a method to employ a trained deep reinforcement learning policy for dedicated high-level behavior planning. By popul… ▽ More Reinforcement learning has received high research interest for developing planning approaches in automated driving. Most prior works consider the end-to-end planning task that yields direct control commands and rarely deploy their algorithm to real vehicles. In this work, we propose a method to employ a trained deep reinforcement learning policy for dedicated high-level behavior planning. By populating an abstract objective interface, established motion planning algorithms can be leveraged, which derive smooth and drivable trajectories. Given the current environment model, we propose to use a built-in simulator to predict the traffic scene for a given horizon into the future. The behavior of automated vehicles in mixed traffic is determined by querying the learned policy. To the best of our knowledge, this work is the first to apply deep reinforcement learning in this manner, and as such lacks a state-of-the-art benchmark. Thus, we validate the proposed approach by comparing an idealistic single-shot plan with cyclic replanning through the learned policy. Experiments with a real testing vehicle on proving grounds demonstrate the potential of our approach to shrink the simulation to real world gap of deep reinforcement learning based planning approaches. Additional simulative analyses reveal that more complex multi-agent maneuvers can be managed by employing the cycling replanning approach. △ Less

Submitted 17 April, 2023; originally announced April 2023.

Comments: 8 pages, 10 figures, to be published in 34th IEEE Intelligent Vehicles Symposium (IV)

The Impact of Frame-Dropping on Performance and Energy Consumption for Multi-Object Tracking

Authors: Matti Henning, Michael Buchholz, Klaus Dietmayer

Abstract: The safety of automated vehicles (AVs) relies on the representation of their environment. Consequently, state-of-the-art AVs employ potent sensor systems to achieve the best possible environment representation at all times. Although these high-performing systems achieve impressive results, they induce significant requirements for the processing capabilities of an AV's computational hardware compon… ▽ More The safety of automated vehicles (AVs) relies on the representation of their environment. Consequently, state-of-the-art AVs employ potent sensor systems to achieve the best possible environment representation at all times. Although these high-performing systems achieve impressive results, they induce significant requirements for the processing capabilities of an AV's computational hardware components and their energy consumption. To enable a dynamic adaptation of such perception systems based on the situational perception requirements, we introduce a model-agnostic method for the scalable employment of single-frame object detection models using frame-dropping in tracking-by-detection systems. We evaluate our approach on the KITTI 3D Tracking Benchmark, showing that significant energy savings can be achieved at acceptable performance degradation, reaching up to 28% reduction of energy consumption at a performance decline of 6.6% in HOTA score. △ Less

Submitted 1 August, 2023; v1 submitted 17 April, 2023; originally announced April 2023.

Comments: V1: Accepted for publication at the IEEE Intelligent Vehicle Symposium, 2023 V2: Added DOI 10.1109/IV55152.2023.10186653

Journal ref: 2023 IEEE Intelligent Vehicles Symposium (IV), Anchorage, AK, USA, 2023, pp. 1-6

Automatic Intersection Management in Mixed Traffic Using Reinforcement Learning and Graph Neural Networks

Authors: Marvin Klimke, Benjamin Völz, Michael Buchholz

Abstract: Connected automated driving has the potential to significantly improve urban traffic efficiency, e.g., by alleviating issues due to occlusion. Cooperative behavior planning can be employed to jointly optimize the motion of multiple vehicles. Most existing approaches to automatic intersection management, however, only consider fully automated traffic. In practice, mixed traffic, i.e., the simultane… ▽ More Connected automated driving has the potential to significantly improve urban traffic efficiency, e.g., by alleviating issues due to occlusion. Cooperative behavior planning can be employed to jointly optimize the motion of multiple vehicles. Most existing approaches to automatic intersection management, however, only consider fully automated traffic. In practice, mixed traffic, i.e., the simultaneous road usage by automated and human-driven vehicles, will be prevalent. The present work proposes to leverage reinforcement learning and a graph-based scene representation for cooperative multi-agent planning. We build upon our previous works that showed the applicability of such machine learning methods to fully automated traffic. The scene representation is extended for mixed traffic and considers uncertainty in the human drivers' intentions. In the simulation-based evaluation, we model measurement uncertainties through noise processes that are tuned using real-world data. The paper evaluates the proposed method against an enhanced first in - first out scheme, our baseline for mixed traffic management. With increasing share of automated vehicles, the learned planner significantly increases the vehicle throughput and reduces the delay due to interaction. Non-automated vehicles benefit virtually alike. △ Less

Submitted 28 July, 2023; v1 submitted 30 January, 2023; originally announced January 2023.

Comments: 8 pages, 7 figures, 34th IEEE Intelligent Vehicles Symposium (IV), updated to accepted version

Extrinsic Camera Calibration with Semantic Segmentation

Authors: Alexander Tsaregorodtsev, Johannes Müller, Jan Strohbeck, Martin Herrmann, Michael Buchholz, Vasileios Belagiannis

Abstract: Monocular camera sensors are vital to intelligent vehicle operation and automated driving assistance and are also heavily employed in traffic control infrastructure. Calibrating the monocular camera, though, is time-consuming and often requires significant manual intervention. In this work, we present an extrinsic camera calibration approach that automatizes the parameter estimation by utilizing s… ▽ More Monocular camera sensors are vital to intelligent vehicle operation and automated driving assistance and are also heavily employed in traffic control infrastructure. Calibrating the monocular camera, though, is time-consuming and often requires significant manual intervention. In this work, we present an extrinsic camera calibration approach that automatizes the parameter estimation by utilizing semantic segmentation information from images and point clouds. Our approach relies on a coarse initial measurement of the camera pose and builds on lidar sensors mounted on a vehicle with high-precision localization to capture a point cloud of the camera environment. Afterward, a mapping between the camera and world coordinate spaces is obtained by performing a lidar-to-camera registration of the semantically segmented sensor data. We evaluate our method on simulated and real-world data to demonstrate low error measurements in the calibration results. Our approach is suitable for infrastructure sensors as well as vehicle sensors, while it does not require motion of the camera platform. △ Less

Submitted 8 August, 2022; originally announced August 2022.

Comments: 7 pages, 3 figures, accepted at the 25th International Conference on Intelligent Transportation Systems (ITSC) 2022

An Enhanced Graph Representation for Machine Learning Based Automatic Intersection Management

Authors: Marvin Klimke, Jasper Gerigk, Benjamin Völz, Michael Buchholz

Abstract: The improvement of traffic efficiency at urban intersections receives strong research interest in the field of automated intersection management. So far, mostly non-learning algorithms like reservation or optimization-based ones were proposed to solve the underlying multi-agent planning problem. At the same time, automated driving functions for a single ego vehicle are increasingly implemented usi… ▽ More The improvement of traffic efficiency at urban intersections receives strong research interest in the field of automated intersection management. So far, mostly non-learning algorithms like reservation or optimization-based ones were proposed to solve the underlying multi-agent planning problem. At the same time, automated driving functions for a single ego vehicle are increasingly implemented using machine learning methods. In this work, we build upon a previously presented graph-based scene representation and graph neural network to approach the problem using reinforcement learning. The scene representation is improved in key aspects by using edge features in addition to the existing node features for the vehicles. This leads to an increased representation quality that is leveraged by an updated network architecture. The paper provides an in-depth evaluation of the proposed method against baselines that are commonly used in automatic intersection management. Compared to a traditional signalized intersection and an enhanced first-in-first-out scheme, a significant reduction of induced delay is observed at varying traffic densities. Finally, the generalization capability of the graph-based representation is evaluated by testing the policy on intersection layouts not seen during training. The model generalizes virtually without restrictions to smaller intersection layouts and within certain limits to larger ones. △ Less

Submitted 18 July, 2022; originally announced July 2022.

Comments: 8 pages, 11 figures, to be published in IEEE 25th International Conference on Intelligent Transportation Systems (ITSC)

Identification of Threat Regions From a Dynamic Occupancy Grid Map for Situation-Aware Environment Perception

Authors: Matti Henning, Jan Strohbeck, Michael Buchholz, Klaus Dietmayer

Abstract: The advance towards higher levels of automation within the field of automated driving is accompanied by increasing requirements for the operational safety of vehicles. Induced by the limitation of computational resources, trade-offs between the computational complexity of algorithms and their potential to ensure safe operation of automated vehicles are often encountered. Situation-aware environmen… ▽ More The advance towards higher levels of automation within the field of automated driving is accompanied by increasing requirements for the operational safety of vehicles. Induced by the limitation of computational resources, trade-offs between the computational complexity of algorithms and their potential to ensure safe operation of automated vehicles are often encountered. Situation-aware environment perception presents one promising example, where computational resources are distributed to regions within the perception area that are relevant for the task of the automated vehicle. While prior map knowledge is often leveraged to identify relevant regions, in this work, we present a lightweight identification of safety-relevant regions that relies solely on online information. We show that our approach enables safe vehicle operation in critical scenarios, while retaining the benefits of non-uniformly distributed resources within the environment perception. △ Less

Submitted 14 February, 2023; v1 submitted 5 July, 2022; originally announced July 2022.

Comments: Accepted for publication at the 25th IEEE International Conference on Intelligent Transportation Systems 2022. V2: added IEEE copyright notice V3: Added DOI

Journal ref: 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), 2022, pp. 805-810

Situation-Aware Environment Perception for Decentralized Automation Architectures

Authors: Matti Henning, Michael Buchholz, Klaus Dietmayer

Abstract: Advances in the field of environment perception for automated agents have resulted in an ongoing increase in generated sensor data. The available computational resources to process these data are bound to become insufficient for real-time applications. Reducing the amount of data to be processed by identifying the most relevant data based on the agents' situation, often referred to as situation-aw… ▽ More Advances in the field of environment perception for automated agents have resulted in an ongoing increase in generated sensor data. The available computational resources to process these data are bound to become insufficient for real-time applications. Reducing the amount of data to be processed by identifying the most relevant data based on the agents' situation, often referred to as situation-awareness, has gained increasing research interest, and the importance of complementary approaches is expected to increase further in the near future. In this work, we extend the applicability range of our recently introduced concept for situation-aware environment perception to the decentralized automation architecture of the UNICARagil project. Considering the specific driving capabilities of the vehicle and using real-world data on target hardware in a post-processing manner, we provide an estimate for the daily reduction in power consumption that accumulates to 36.2%. While achieving these promising results, we additionally show the need to consider scalability in data processing in the design of software modules as well as in the design of functional systems if the benefits of situation-awareness shall be leveraged optimally. △ Less

Submitted 26 July, 2022; v1 submitted 5 July, 2022; originally announced July 2022.

Comments: V2: added DOI to copyright notice

Journal ref: 2022 IEEE Intelligent Vehicles Symposium (IV), 2022, pp. 1087-1092

MGNet: Monocular Geometric Scene Understanding for Autonomous Driving

Authors: Markus Schön, Michael Buchholz, Klaus Dietmayer

Abstract: We introduce MGNet, a multi-task framework for monocular geometric scene understanding. We define monocular geometric scene understanding as the combination of two known tasks: Panoptic segmentation and self-supervised monocular depth estimation. Panoptic segmentation captures the full scene not only semantically, but also on an instance basis. Self-supervised monocular depth estimation uses geome… ▽ More We introduce MGNet, a multi-task framework for monocular geometric scene understanding. We define monocular geometric scene understanding as the combination of two known tasks: Panoptic segmentation and self-supervised monocular depth estimation. Panoptic segmentation captures the full scene not only semantically, but also on an instance basis. Self-supervised monocular depth estimation uses geometric constraints derived from the camera measurement model in order to measure depth from monocular video sequences only. To the best of our knowledge, we are the first to propose the combination of these two tasks in one single model. Our model is designed with focus on low latency to provide fast inference in real-time on a single consumer-grade GPU. During deployment, our model produces dense 3D point clouds with instance aware semantic labels from single high-resolution camera images. We evaluate our model on two popular autonomous driving benchmarks, i.e., Cityscapes and KITTI, and show competitive performance among other real-time capable methods. Source code is available at https://github.com/markusschoen/MGNet. △ Less

Submitted 27 June, 2022; originally announced June 2022.

Journal ref: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 15784-15795

Self-Assessment for Single-Object Tracking in Clutter Using Subjective Logic

Authors: Thomas Griebel, Johannes Müller, Paul Geisler, Charlotte Hermann, Martin Herrmann, Michael Buchholz, Klaus Dietmayer

Abstract: Reliable tracking algorithms are essential for automated driving. However, the existing consistency measures are not sufficient to meet the increasing safety demands in the automotive sector. Therefore, this work presents a novel method for self-assessment of single-object tracking in clutter based on Kalman filtering and subjective logic. A key feature of the approach is that it additionally prov… ▽ More Reliable tracking algorithms are essential for automated driving. However, the existing consistency measures are not sufficient to meet the increasing safety demands in the automotive sector. Therefore, this work presents a novel method for self-assessment of single-object tracking in clutter based on Kalman filtering and subjective logic. A key feature of the approach is that it additionally provides a measure of the collected statistical evidence in its online reliability scores. In this way, various aspects of reliability, such as the correctness of the assumed measurement noise, detection probability, and clutter rate, can be monitored in addition to the overall assessment based on the available evidence. Here, we present a mathematical derivation of the reference distribution used in our self-assessment module for our studied problem. Moreover, we introduce a formula that describes how a threshold should be chosen for the degree of conflict, the subjective logic comparison measure used for the reliability decision making. Our approach is evaluated in a challenging simulation scenario designed to model adverse weather conditions. The simulations show that our method can significantly improve the reliability checking of single-object tracking in clutter in several aspects. △ Less

Submitted 15 June, 2022; originally announced June 2022.

Comments: Accepted for presentation at the 2022 IEEE 25th International Conference on Information Fusion (FUSION), July 4 - 7, 2022, Linköping, Sweden

Cooperative Behavior Planning for Automated Driving using Graph Neural Networks

Authors: Marvin Klimke, Benjamin Völz, Michael Buchholz

Abstract: Urban intersections are prone to delays and inefficiencies due to static precedence rules and occlusions limiting the view on prioritized traffic. Existing approaches to improve traffic flow, widely known as automatic intersection management systems, are mostly based on non-learning reservation schemes or optimization algorithms. Machine learning-based techniques show promising results in planning… ▽ More Urban intersections are prone to delays and inefficiencies due to static precedence rules and occlusions limiting the view on prioritized traffic. Existing approaches to improve traffic flow, widely known as automatic intersection management systems, are mostly based on non-learning reservation schemes or optimization algorithms. Machine learning-based techniques show promising results in planning for a single ego vehicle. This work proposes to leverage machine learning algorithms to optimize traffic flow at urban intersections by jointly planning for multiple vehicles. Learning-based behavior planning poses several challenges, demanding for a suited input and output representation as well as large amounts of ground-truth data. We address the former issue by using a flexible graph-based input representation accompanied by a graph neural network. This allows to efficiently encode the scene and inherently provide individual outputs for all involved vehicles. To learn a sensible policy, without relying on the imitation of expert demonstrations, the cooperative planning task is considered as a reinforcement learning problem. We train and evaluate the proposed method in an open-source simulation environment for decision making in automated driving. Compared to a first-in-first-out scheme and traffic governed by static priority rules, the learned planner shows a significant gain in flow rate, while reducing the number of induced stops. In addition to synthetic simulations, the approach is also evaluated based on real-world traffic data taken from the publicly available inD dataset. △ Less

Submitted 26 July, 2022; v1 submitted 23 February, 2022; originally announced February 2022.

Comments: 8 pages, 8 figures, 33rd IEEE Intelligent Vehicles Symposium (IV), updated to accepted version

Globally Optimal Multi-Scale Monocular Hand-Eye Calibration Using Dual Quaternions

Authors: Thomas Wodtko, Markus Horn, Michael Buchholz, Klaus Dietmayer

Abstract: In this work, we present an approach for monocular hand-eye calibration from per-sensor ego-motion based on dual quaternions. Due to non-metrically scaled translations of monocular odometry, a scaling factor has to be estimated in addition to the rotation and translation calibration. For this, we derive a quadratically constrained quadratic program that allows a combined estimation of all extrinsi… ▽ More In this work, we present an approach for monocular hand-eye calibration from per-sensor ego-motion based on dual quaternions. Due to non-metrically scaled translations of monocular odometry, a scaling factor has to be estimated in addition to the rotation and translation calibration. For this, we derive a quadratically constrained quadratic program that allows a combined estimation of all extrinsic calibration parameters. Using dual quaternions leads to low run-times due to their compact representation. Our problem formulation further allows to estimate multiple scalings simultaneously for different sequences of the same sensor setup. Based on our problem formulation, we derive both, a fast local and a globally optimal solving approach. Finally, our algorithms are evaluated and compared to state-of-the-art approaches on simulated and real-world data, e.g., the EuRoC MAV dataset. △ Less

Submitted 12 January, 2022; originally announced January 2022.

Journal ref: 2021 International Conference on 3D Vision (3DV)

Situation-Aware Environment Perception Using a Multi-Layer Attention Map

Authors: Matti Henning, Johannes Müller, Fabian Gies, Michael Buchholz, Klaus Dietmayer

Abstract: Within the field of automated driving, a clear trend in environment perception tends towards more sensors, higher redundancy, and overall increase in computational power. This is mainly driven by the paradigm to perceive the entire environment as best as possible at all times. However, due to the ongoing rise in functional complexity, compromises have to be considered to ensure real-time capabilit… ▽ More Within the field of automated driving, a clear trend in environment perception tends towards more sensors, higher redundancy, and overall increase in computational power. This is mainly driven by the paradigm to perceive the entire environment as best as possible at all times. However, due to the ongoing rise in functional complexity, compromises have to be considered to ensure real-time capabilities of the perception system. In this work, we introduce a concept for situation-aware environment perception to control the resource allocation towards processing relevant areas within the data as well as towards employing only a subset of functional modules for environment perception, if sufficient for the current driving task. Specifically, we propose to evaluate the context of an automated vehicle to derive a multi-layer attention map (MLAM) that defines relevant areas. Using this MLAM, the optimum of active functional modules is dynamically configured and intra-module processing of only relevant data is enforced. We outline the feasibility of application of our concept using real-world data in a straight-forward implementation for our system at hand. While retaining overall functionality, we achieve a reduction of accumulated processing time of 59%. △ Less

Submitted 4 April, 2022; v1 submitted 2 December, 2021; originally announced December 2021.

Comments: Update 1: Removed minor typos, corrected FKZ. Update 2: Extension of Section III-E. Rewording in some sections to improve clarity. Added IEEE Copyright Notice

Report number: Volume: 8, Issue: 1, January 2023

Journal ref: IEEE Transactions on Intelligent Vehicles, 2023

Motion Planning for Connected Automated Vehicles at Occluded Intersections With Infrastructure Sensors

Authors: Johannes Müller, Jan Strohbeck, Martin Herrmann, Michael Buchholz

Abstract: Motion planning at urban intersections that accounts for the situation context, handles occlusions, and deals with measurement and prediction uncertainty is a major challenge on the way to urban automated driving. In this work, we address this challenge with a sampling-based optimization approach. For this, we formulate an optimal control problem that optimizes for low risk and high passenger comf… ▽ More Motion planning at urban intersections that accounts for the situation context, handles occlusions, and deals with measurement and prediction uncertainty is a major challenge on the way to urban automated driving. In this work, we address this challenge with a sampling-based optimization approach. For this, we formulate an optimal control problem that optimizes for low risk and high passenger comfort. The risk is calculated on the basis of the perception information and the respective uncertainty using a risk model. The risk model combines set-based methods and probabilistic approaches. Thus, the approach provides safety guarantees in a probabilistic sense, while for a vanishing risk, the formal safety guarantees of the set-based methods are inherited. By exploring all available behavior options, our approach solves decision making and longitudinal trajectory planning in one step. The available behavior options are provided by a formal representation of the situation context, which is also used to reduce calculation efforts. Occlusions are resolved using the external perception of infrastructure-mounted sensors. Yet, instead of merging external and ego perception with track-to-track fusion, the information is used in parallel. The motion planning scheme is validated through real-world experiments. △ Less

Submitted 21 October, 2021; originally announced October 2021.

Comments: 12 pages, 8 figures

Anomaly Detection in Radar Data Using PointNets

Authors: Thomas Griebel, Dominik Authaler, Markus Horn, Matti Henning, Michael Buchholz, Klaus Dietmayer

Abstract: For autonomous driving, radar is an important sensor type. On the one hand, radar offers a direct measurement of the radial velocity of targets in the environment. On the other hand, in literature, radar sensors are known for their robustness against several kinds of adverse weather conditions. However, on the downside, radar is susceptible to ghost targets or clutter which can be caused by severa… ▽ More For autonomous driving, radar is an important sensor type. On the one hand, radar offers a direct measurement of the radial velocity of targets in the environment. On the other hand, in literature, radar sensors are known for their robustness against several kinds of adverse weather conditions. However, on the downside, radar is susceptible to ghost targets or clutter which can be caused by several different causes, e.g., reflective surfaces in the environment. Ghost targets, for instance, can result in erroneous object detections. To this end, it is desirable to identify anomalous targets as early as possible in radar data. In this work, we present an approach based on PointNets to detect anomalous radar targets. Modifying the PointNet-architecture driven by our task, we developed a novel grouping variant which contributes to a multi-form grouping module. Our method is evaluated on a real-world dataset in urban scenarios and shows promising results for the detection of anomalous radar targets. △ Less

Submitted 20 September, 2021; originally announced September 2021.

Comments: Accepted for presentation at the 2021 IEEE 24th International Conference on Intelligent Transportation Systems (ITSC), September 19-22, 2021, Indianapolis, USA

Online Extrinsic Calibration based on Per-Sensor Ego-Motion Using Dual Quaternions

Authors: Markus Horn, Thomas Wodtko, Michael Buchholz, Klaus Dietmayer

Abstract: In this work, we propose an approach for extrinsic sensor calibration from per-sensor ego-motion estimates. Our problem formulation is based on dual quaternions, enabling two different online capable solving approaches. We provide a certifiable globally optimal and a fast local approach along with a method to verify the globality of the local approach. Additionally, means for integrating previous… ▽ More In this work, we propose an approach for extrinsic sensor calibration from per-sensor ego-motion estimates. Our problem formulation is based on dual quaternions, enabling two different online capable solving approaches. We provide a certifiable globally optimal and a fast local approach along with a method to verify the globality of the local approach. Additionally, means for integrating previous knowledge, for example, a common ground plane for planar sensor motion, are described. Our algorithms are evaluated on simulated data and on a publicly available dataset containing RGB-D camera images. Further, our online calibration approach is tested on the KITTI odometry dataset, which provides data of a lidar and two stereo camera systems mounted on a vehicle. Our evaluation confirms the short run time, state-of-the-art accuracy, as well as online capability of our approach while retaining the global optimality of the solution at any time. △ Less

Submitted 1 March, 2021; v1 submitted 27 January, 2021; originally announced January 2021.

Comments: 8 pages, 6 figures, 2 tables, published in RA-L

Journal ref: IEEE Robotics and Automation Letters 6 (2), 982-989, 2021

LMB Filter Based Tracking Allowing for Multiple Hypotheses in Object Reference Point Association*

Authors: Martin Herrmann, Aldi Piroli, Jan Strohbeck, Johannes Müller, Michael Buchholz

Abstract: Autonomous vehicles need precise knowledge on dynamic objects in their surroundings. Especially in urban areas with many objects and possible occlusions, an infrastructure system based on a multi-sensor setup can provide the required environment model for the vehicles. Previously, we have published a concept of object reference points (e.g. the corners of an object), which allows for generic senso… ▽ More Autonomous vehicles need precise knowledge on dynamic objects in their surroundings. Especially in urban areas with many objects and possible occlusions, an infrastructure system based on a multi-sensor setup can provide the required environment model for the vehicles. Previously, we have published a concept of object reference points (e.g. the corners of an object), which allows for generic sensor "plug and play" interfaces and relatively cheap sensors. This paper describes a novel method to additionally incorporate multiple hypotheses for fusing the measurements of the object reference points using an extension to the previously presented Labeled Multi-Bernoulli (LMB) filter. In contrast to the previous work, this approach improves the tracking quality in the cases where the correct association of the measurement and the object reference point is unknown. Furthermore, this paper identifies options based on physical models to sort out inconsistent and unfeasible associations at an early stage in order to keep the method computationally tractable for real-time applications. The method is evaluated on simulations as well as on real scenarios. In comparison to comparable methods, the proposed approach shows a considerable performance increase, especially the number of non-continuous tracks is decreased significantly. △ Less

Submitted 11 November, 2020; originally announced November 2020.

Journal ref: 2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI)

DeepCLR: Correspondence-Less Architecture for Deep End-to-End Point Cloud Registration

Authors: Markus Horn, Nico Engel, Vasileios Belagiannis, Michael Buchholz, Klaus Dietmayer

Abstract: This work addresses the problem of point cloud registration using deep neural networks. We propose an approach to predict the alignment between two point clouds with overlapping data content, but displaced origins. Such point clouds originate, for example, from consecutive measurements of a LiDAR mounted on a moving platform. The main difficulty in deep registration of raw point clouds is the fusi… ▽ More This work addresses the problem of point cloud registration using deep neural networks. We propose an approach to predict the alignment between two point clouds with overlapping data content, but displaced origins. Such point clouds originate, for example, from consecutive measurements of a LiDAR mounted on a moving platform. The main difficulty in deep registration of raw point clouds is the fusion of template and source point cloud. Our proposed architecture applies flow embedding to tackle this problem, which generates features that describe the motion of each template point. These features are then used to predict the alignment in an end-to-end fashion without extracting explicit point correspondences between both input clouds. We rely on the KITTI odometry and ModelNet40 datasets for evaluating our method on various point distributions. Our approach achieves state-of-the-art accuracy and the lowest run-time of the compared methods. △ Less

Submitted 13 January, 2021; v1 submitted 22 July, 2020; originally announced July 2020.

Comments: 7 pages, 5 figures, 4 tables

Journal ref: 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)

Kalman Filter Meets Subjective Logic: A Self-Assessing Kalman Filter Using Subjective Logic

Authors: Thomas Griebel, Johannes Müller, Michael Buchholz, Klaus Dietmayer

Abstract: Self-assessment is a key to safety and robustness in automated driving. In order to design safer and more robust automated driving functions, the goal is to self-assess the performance of each module in a whole automated driving system. One crucial component in automated driving systems is the tracking of surrounding objects, where the Kalman filter is the most fundamental tracking algorithm. For… ▽ More Self-assessment is a key to safety and robustness in automated driving. In order to design safer and more robust automated driving functions, the goal is to self-assess the performance of each module in a whole automated driving system. One crucial component in automated driving systems is the tracking of surrounding objects, where the Kalman filter is the most fundamental tracking algorithm. For Kalman filters, some classical online consistency measures exist for self-assessment, which are based on classical probability theory. However, these classical approaches lack the ability to measure the explicit statistical uncertainty within the self-assessment, which is an important quality measure, particularly, if only a small number of samples is available for the self-assessment. In this work, we propose a novel online self-assessment method using subjective logic, which is a modern extension of probabilistic logic that explicitly models the statistical uncertainty. Thus, by embedding classical Kalman filtering into subjective logic, our method additionally features an explicit measure for statistical uncertainty in the self-assessment. △ Less

Submitted 1 July, 2020; originally announced July 2020.

Comments: Accepted for presentation at the 2020 IEEE 23rd International Conference on Information Fusion (FUSION), July 6 - 9, 2020, Virtual Conference

Extended Existence Probability Using Digital Maps for Object Verification

Authors: Fabian Gies, Joachim Posselt, Michael Buchholz, Klaus Dietmayer

Abstract: A main task for automated vehicles is an accurate and robust environment perception. Especially, an error-free detection and modeling of other traffic participants is of great importance to drive safely in any situation. For this purpose, multi-object tracking algorithms, based on object detections from raw sensor measurements, are commonly used. However, false object hypotheses can occur due to a… ▽ More A main task for automated vehicles is an accurate and robust environment perception. Especially, an error-free detection and modeling of other traffic participants is of great importance to drive safely in any situation. For this purpose, multi-object tracking algorithms, based on object detections from raw sensor measurements, are commonly used. However, false object hypotheses can occur due to a high density of different traffic participants in complex, arbitrary scenarios. For this reason, the presented approach introduces a probabilistic model to verify the existence of a tracked object. Therefore, an object verification module is introduced, where the influences of multiple digital map elements on a track's existence are evaluated. Finally, a probabilistic model fuses the various influences and estimates an extended existence probability for every track. In addition, a Bayes Net is implemented as directed graphical model to highlight this work's expandability. The presented approach, reduces the number of false positives, while retaining true positives. Real world data is used to evaluate and to highlight the benefits of the presented approach, especially in urban scenarios. △ Less

Submitted 29 May, 2020; v1 submitted 23 March, 2020; originally announced March 2020.

LACI: Low-effort Automatic Calibration of Infrastructure Sensors

Authors: Johannes Müller, Martin Herrmann, Jan Strohbeck, Vasileios Belagiannis, Michael Buchholz

Abstract: Sensor calibration usually is a time consuming yet important task. While classical approaches are sensor-specific and often need calibration targets as well as a widely overlapping field of view (FOV), within this work, a cooperative intelligent vehicle is used as callibration target. The vehicleis detected in the sensor frame and then matched with the information received from the cooperative awa… ▽ More Sensor calibration usually is a time consuming yet important task. While classical approaches are sensor-specific and often need calibration targets as well as a widely overlapping field of view (FOV), within this work, a cooperative intelligent vehicle is used as callibration target. The vehicleis detected in the sensor frame and then matched with the information received from the cooperative awareness messagessend by the coperative intelligent vehicle. The presented algorithm is fully automated as well as sensor-independent, relying only on a very common set of assumptions. Due to the direct registration on the world frame, no overlapping FOV is necessary. The algorithm is evaluated through experiment for four laserscanners as well as one pair of stereo cameras showing a repetition error within the measurement uncertainty of the sensors. A plausibility check rules out systematic errors that might not have been covered by evaluating the repetition error. △ Less

Submitted 5 November, 2019; originally announced November 2019.

Comments: 6 pages, published at ITSC 2019

A Trust Management and Misbehaviour Detection Mechanism for Multi-Agent Systems and its Application to Intelligent Transportation Systems

Authors: Johannes Müller, Tobias Meuser, Ralf Steinmetz, Michael Buchholz

Abstract: Cooperative information shared among a multi-agent system (MAS) can be useful to agents to efficiently fulfill their missions. Relying on wrong information, however, can have severe consequences. While classical approaches only consider measurement uncertainty, reliability information on the incoming data can be useful for decision making. In this work, a subjective logic based mechanism is propos… ▽ More Cooperative information shared among a multi-agent system (MAS) can be useful to agents to efficiently fulfill their missions. Relying on wrong information, however, can have severe consequences. While classical approaches only consider measurement uncertainty, reliability information on the incoming data can be useful for decision making. In this work, a subjective logic based mechanism is proposed that amends reliability information to the data shared among the MAS. If multiple agents report the same event, their information is fused. In order to maintain high reliability, the mechanism detects and isolates misbehaving agents. Therefore, an attacker model is specified that includes faulty as well as malicious agents. The mechanism is applied to Intelligent Transportation Systems (ITS) and it is shown in simulation that the approach scales well with the size of the MAS and that it is able to efficiently detected and isolated misbehaving agents. Keywords: Multi-agent systems, Fault Detection, Sensor/data fusion, Control Applications △ Less

Submitted 22 May, 2019; originally announced May 2019.

Comments: 7 pages, accepted on 15th IEEE International Conference on Control and Automation (IEEE ICCA 2019)

A Subjective-Logic-based Reliability Estimation Mechanism for Cooperative Information with Application to IV's Safety

Authors: Johannes Müller, Michael Gabb, Michael Buchholz

Abstract: Use of cooperative information, distributed by road-side units, offers large potential for intelligent vehicles (IVs). As vehicle automation progresses and cooperative perception is used to fill the blind spots of onboard sensors, the question of reliability of the data becomes increasingly important in safety considerations (SOTIF, Safety of the Intended Functionality). This paper addresses the… ▽ More Use of cooperative information, distributed by road-side units, offers large potential for intelligent vehicles (IVs). As vehicle automation progresses and cooperative perception is used to fill the blind spots of onboard sensors, the question of reliability of the data becomes increasingly important in safety considerations (SOTIF, Safety of the Intended Functionality). This paper addresses the problem to estimate the reliability of cooperative information for in-vehicle use. We propose a novel method to infer the reliability of received data based on the theory of Subjective Logic (SL). Using SL, we fuse multiple information sources, which individually only provide mild cues of the reliability, into a holistic estimate, which is statistically sound through an end-to-end modeling within the theory of SL. Using the proposed scheme for probabilistic SL-based fusion, IVs are able to separate faulty from correct data samples with a large margin of safety. Real world experiments show the applicability and effectiveness of our approach. △ Less

Submitted 22 May, 2019; v1 submitted 12 February, 2019; originally announced March 2019.

Comments: 7 pages, accepted at 30th IEEE Intelligent Vehicles Symposium (IV 2019)

ls1 mardyn: The massively parallel molecular dynamics code for large systems

Authors: Christoph Niethammer, Stefan Becker, Martin Bernreuther, Martin Buchholz, Wolfgang Eckhardt, Alexander Heinecke, Stephan Werth, Hans-Joachim Bungartz, Colin W. Glass, Hans Hasse, Jadran Vrabec, Martin Horsch

Abstract: The molecular dynamics simulation code ls1 mardyn is presented. It is a highly scalable code, optimized for massively parallel execution on supercomputing architectures, and currently holds the world record for the largest molecular simulation with over four trillion particles. It enables the application of pair potentials to length and time scales which were previously out of scope for molecular… ▽ More The molecular dynamics simulation code ls1 mardyn is presented. It is a highly scalable code, optimized for massively parallel execution on supercomputing architectures, and currently holds the world record for the largest molecular simulation with over four trillion particles. It enables the application of pair potentials to length and time scales which were previously out of scope for molecular dynamics simulation. With an efficient dynamic load balancing scheme, it delivers high scalability even for challenging heterogeneous configurations. Presently, multi-center rigid potential models based on Lennard-Jones sites, point charges and higher-order polarities are supported. Due to its modular design, ls1 mardyn can be extended to new physical models, methods, and algorithms, allowing future users to tailor it to suit their respective needs. Possible applications include scenarios with complex geometries, e.g. for fluids at interfaces, as well as non-equilibrium molecular dynamics simulation of heat and mass transfer. △ Less

Submitted 20 August, 2014; originally announced August 2014.