-
3D Face Reconstruction From Radar Images
Authors:
Valentin Braeutigam,
Vanessa Wirth,
Ingrid Ullmann,
Christian Schüßler,
Martin Vossiek,
Matthias Berking,
Bernhard Egger
Abstract:
The 3D reconstruction of faces gains wide attention in computer vision and is used in many fields of application, for example, animation, virtual reality, and even forensics. This work is motivated by monitoring patients in sleep laboratories. Due to their unique characteristics, sensors from the radar domain have advantages compared to optical sensors, namely penetration of electrically non-condu…
▽ More
The 3D reconstruction of faces gains wide attention in computer vision and is used in many fields of application, for example, animation, virtual reality, and even forensics. This work is motivated by monitoring patients in sleep laboratories. Due to their unique characteristics, sensors from the radar domain have advantages compared to optical sensors, namely penetration of electrically non-conductive materials and independence of light. These advantages of radar signals unlock new applications and require adaptation of 3D reconstruction frameworks. We propose a novel model-based method for 3D reconstruction from radar images. We generate a dataset of synthetic radar images with a physics-based but non-differentiable radar renderer. This dataset is used to train a CNN-based encoder to estimate the parameters of a 3D morphable face model. Whilst the encoder alone already leads to strong reconstructions of synthetic data, we extend our reconstruction in an Analysis-by-Synthesis fashion to a model-based autoencoder. This is enabled by learning the rendering process in the decoder, which acts as an object-specific differentiable radar renderer. Subsequently, the combination of both network parts is trained to minimize both, the loss of the parameters and the loss of the resulting reconstructed radar image. This leads to the additional benefit, that at test time the parameters can be further optimized by finetuning the autoencoder unsupervised on the image loss. We evaluated our framework on generated synthetic face images as well as on real radar images with 3D ground truth of four individuals.
△ Less
Submitted 5 February, 2025; v1 submitted 3 December, 2024;
originally announced December 2024.
-
MAROON: A Framework for the Joint Characterization of Near-Field High-Resolution Radar and Optical Depth Imaging Techniques
Authors:
Vanessa Wirth,
Johanna Bräunig,
Martin Vossiek,
Tim Weyrich,
Marc Stamminger
Abstract:
Utilizing the complementary strengths of wavelength-specific range or depth sensors is crucial for robust computer-assisted tasks such as autonomous driving. Despite this, there is still little research done at the intersection of optical depth sensors and radars operating close range, where the target is decimeters away from the sensors. Together with a growing interest in high-resolution imaging…
▽ More
Utilizing the complementary strengths of wavelength-specific range or depth sensors is crucial for robust computer-assisted tasks such as autonomous driving. Despite this, there is still little research done at the intersection of optical depth sensors and radars operating close range, where the target is decimeters away from the sensors. Together with a growing interest in high-resolution imaging radars operating in the near field, the question arises how these sensors behave in comparison to their traditional optical counterparts.
In this work, we take on the unique challenge of jointly characterizing depth imagers from both, the optical and radio-frequency domain using a multimodal spatial calibration. We collect data from four depth imagers, with three optical sensors of varying operation principle and an imaging radar. We provide a comprehensive evaluation of their depth measurements with respect to distinct object materials, geometries, and object-to-sensor distances. Specifically, we reveal scattering effects of partially transmissive materials and investigate the response of radio-frequency signals. All object measurements will be made public in form of a multimodal dataset, called MAROON.
△ Less
Submitted 26 November, 2024; v1 submitted 1 November, 2024;
originally announced November 2024.
-
Sensing Accuracy Optimization for Communication-assisted Dual-baseline UAV-InSAR
Authors:
Mohamed-Amine Lahmeri,
Víctor Mustieles-Pérez,
Martin Vossiek,
Gerhard Krieger,
Robert Schober
Abstract:
In this paper, we study the optimization of the sensing accuracy of unmanned aerial vehicle (UAV)-based dual-baseline interferometric synthetic aperture radar (InSAR) systems. A swarm of three UAV-synthetic aperture radar (SAR) systems is deployed to image an area of interest from different angles, enabling the creation of two independent digital elevation models (DEMs). To reduce the InSAR sensin…
▽ More
In this paper, we study the optimization of the sensing accuracy of unmanned aerial vehicle (UAV)-based dual-baseline interferometric synthetic aperture radar (InSAR) systems. A swarm of three UAV-synthetic aperture radar (SAR) systems is deployed to image an area of interest from different angles, enabling the creation of two independent digital elevation models (DEMs). To reduce the InSAR sensing error, i.e., the height estimation error, the two DEMs are fused based on weighted averaging techniques into one final DEM. The heavy computations required for this process are performed on the ground. To this end, the radar data is offloaded in real time via a frequency division multiple access (FDMA) air-to-ground backhaul link. In this work, we focus on improving the sensing accuracy by minimizing the worst-case height estimation error of the final DEM. To this end, the UAV formation and the power allocated for offloading are jointly optimized based on alternating optimization (AO), while meeting practical InSAR sensing and communication constraints. Our simulation results demonstrate that the proposed solution can significantly improve the sensing accuracy compared to classical single-baseline UAV-InSAR systems and other benchmark schemes.
△ Less
Submitted 2 November, 2024; v1 submitted 24 October, 2024;
originally announced October 2024.
-
UAV Formation and Resource Allocation Optimization for Communication-Assisted 3D InSAR Sensing
Authors:
Mohamed-Amine Lahmeri,
Víctor Mustieles-Pérez,
Martin Vossiek,
Gerhard Krieger,
Robert Schober
Abstract:
In this paper, we investigate joint unmanned aerial vehicle (UAV) formation and resource allocation optimization for communication-assisted three-dimensional (3D) synthetic aperture radar (SAR) sensing. We consider a system consisting of two UAVs that perform bistatic interferometric SAR (InSAR) sensing for generation of a digital elevation model (DEM) and transmit the radar raw data to a ground s…
▽ More
In this paper, we investigate joint unmanned aerial vehicle (UAV) formation and resource allocation optimization for communication-assisted three-dimensional (3D) synthetic aperture radar (SAR) sensing. We consider a system consisting of two UAVs that perform bistatic interferometric SAR (InSAR) sensing for generation of a digital elevation model (DEM) and transmit the radar raw data to a ground station (GS) in real time. To account for practical 3D sensing requirements, we use non-conventional sensing performance metrics, such as the SAR interferometric coherence, i.e., the local cross-correlation between the two co-registered UAV SAR images, the point-to-point InSAR relative height error, and the height of ambiguity, which together characterize the accuracy with which the InSAR system can determine the height of ground targets. Our objective is to jointly optimize the UAV formation, speed, and communication power allocation for maximization of the InSAR coverage while satisfying energy, communication, and InSAR-specific sensing constraints. To solve the formulated non-smooth and non-convex optimization problem, we divide it into three sub-problems and propose a novel alternating optimization (AO) framework that is based on classical, monotonic, and stochastic optimization techniques. The effectiveness of the proposed algorithm is validated through extensive simulations and compared to several benchmark schemes. Furthermore, our simulation results highlight the impact of the UAV-GS communication link on the flying formation and sensing performance and show that the DEM of a large area of interest can be mapped and offloaded to ground successfully, while the ground topography can be estimated with centimeter-scale precision. Lastly, we demonstrate that a low UAV velocity is preferable for InSAR applications as it leads to better sensing accuracy.
△ Less
Submitted 7 February, 2025; v1 submitted 9 July, 2024;
originally announced July 2024.
-
An Efficient yet High-Performance Method for Precise Radar-Based Imaging of Human Hand Poses
Authors:
Johanna Bräunig,
Vanessa Wirth,
Marc Stamminger,
Ingrid Ullmann,
Martin Vossiek
Abstract:
Contactless hand pose estimation requires sensors that provide precise spatial information and low computational complexity for real-time processing. Unlike vision-based systems, radar offers lighting independence and direct motion assessments. Yet, there is limited research balancing real-time constraints, suitable frame rates for motion evaluations, and the need for precise 3D data. To address t…
▽ More
Contactless hand pose estimation requires sensors that provide precise spatial information and low computational complexity for real-time processing. Unlike vision-based systems, radar offers lighting independence and direct motion assessments. Yet, there is limited research balancing real-time constraints, suitable frame rates for motion evaluations, and the need for precise 3D data. To address this, we extend the ultra-efficient two-tone hand imaging method from our prior work to a three-tone approach. Maintaining high frame rates and real-time constraints, this approach significantly enhances reconstruction accuracy and precision. We assess these measures by evaluating reconstruction results for different hand poses obtained by an imaging radar. Accuracy is assessed against ground truth from a spatially calibrated photogrammetry setup, while precision is measured using 3D-printed hand poses. The results emphasize the method's great potential for future radar-based hand sensing.
△ Less
Submitted 19 June, 2024;
originally announced June 2024.
-
Tight Non-Radiating Bends of 3D-Printed Dielectric Image Lines Based on Electromagnetic Bandgap Mirrors
Authors:
Leonhard Hahn,
Tobias Bader,
Christian Carlowitz,
Martin Vossiek,
Gerald Gold
Abstract:
This paper reports on a novel compact, low-loss bending technique for additively manufactured dielectric image lines between 140 GHz and 220 GHz. Conventional bending approaches require either large curvature radii or technologically challenging permittivity variations to prevent radiation loss at line discontinuities, e.g. caused by narrow bends. In contrast, this work uses extremely compact, eas…
▽ More
This paper reports on a novel compact, low-loss bending technique for additively manufactured dielectric image lines between 140 GHz and 220 GHz. Conventional bending approaches require either large curvature radii or technologically challenging permittivity variations to prevent radiation loss at line discontinuities, e.g. caused by narrow bends. In contrast, this work uses extremely compact, easy-to-manufacture electromagnetic bandgap (EBG) cells to solve the afore mentioned challenge for the first time. These offer excellent reflection properties and thus enable broadband and low-loss (IL < 1 dB) guidance of electromagnetic waves by means of total reflection. Without increasing the complexity of the process, both the high-pass behaviour and the enormous space requirement of conventional dielectric bends are completely avoided. In addition, the use of EBGs improves the mutual isolation of dielectric image lines of up to 30 dB. Therefore, a promising solution for the realization of narrow, 3D-printed, low-loss signal distribution networks in the sub-THz domain is offered.
△ Less
Submitted 7 March, 2024;
originally announced May 2024.
-
Automatic Spatial Calibration of Near-Field MIMO Radar With Respect to Optical Depth Sensors
Authors:
Vanessa Wirth,
Johanna Bräunig,
Danti Khouri,
Florian Gutsche,
Martin Vossiek,
Tim Weyrich,
Marc Stamminger
Abstract:
Despite an emerging interest in MIMO radar, the utilization of its complementary strengths in combination with optical depth sensors has so far been limited to far-field applications, due to the challenges that arise from mutual sensor calibration in the near field. In fact, most related approaches in the autonomous industry propose target-based calibration methods using corner reflectors that hav…
▽ More
Despite an emerging interest in MIMO radar, the utilization of its complementary strengths in combination with optical depth sensors has so far been limited to far-field applications, due to the challenges that arise from mutual sensor calibration in the near field. In fact, most related approaches in the autonomous industry propose target-based calibration methods using corner reflectors that have proven to be unsuitable for the near field. In contrast, we propose a novel, joint calibration approach for optical RGB-D sensors and MIMO radars that is designed to operate in the radar's near-field range, within decimeters from the sensors. Our pipeline consists of a bespoke calibration target, allowing for automatic target detection and localization, followed by the spatial calibration of the two sensor coordinate systems through target registration. We validate our approach using two different depth sensing technologies from the optical domain. The experiments show the efficiency and accuracy of our calibration for various target displacements, as well as its robustness of our localization in terms of signal ambiguities.
△ Less
Submitted 13 August, 2024; v1 submitted 16 March, 2024;
originally announced March 2024.
-
3D-Printed Dielectric Image Lines towards Chip-to-Chip Interconnects for subTHz-Applications
Authors:
Leonhard Hahn,
Tim Pfahler,
Tobias Bader,
Gerald Gold,
Martin Vossiek,
Christian Carlowitz
Abstract:
This paper reports on 3D-printed dielectric image lines for low-loss subTHz applications between 140 and 220 GHz. In contrast to conventional dielectric waveguides, a conductive copper substrate is used to achieve robust routing and increased mechanical stability. For easy integration and characterization of the dielectric image line within a waveguide measurement setup, a low-loss mode-converter…
▽ More
This paper reports on 3D-printed dielectric image lines for low-loss subTHz applications between 140 and 220 GHz. In contrast to conventional dielectric waveguides, a conductive copper substrate is used to achieve robust routing and increased mechanical stability. For easy integration and characterization of the dielectric image line within a waveguide measurement setup, a low-loss mode-converter for flexible mounting is further designed. The characterized overall system exhibits a broadband match of at least 20 dB over the entire frequency band, with minimal losses of below 0.35 dB/cm. Furthermore, multi-line characterization is performed for de-embedding the propagation parameters α and \b{eta} of both the dielectric transmission line and the mode-converter, and finally, the influence of discontinuities such as bending radii on the transmission behavior is evaluated. Due to the simplicity of the underlying 3D-printing technology, the proposed concept features extremely low cost and complexity, yet offers high flexibility and outperforms the losses of conventional transmission lines.
△ Less
Submitted 6 March, 2024;
originally announced March 2024.
-
Radar-Based Recognition of Static Hand Gestures in American Sign Language
Authors:
Christian Schuessler,
Wenxuan Zhang,
Johanna Bräunig,
Marcel Hoffmann,
Michael Stelzig,
Martin Vossiek
Abstract:
In the fast-paced field of human-computer interaction (HCI) and virtual reality (VR), automatic gesture recognition has become increasingly essential. This is particularly true for the recognition of hand signs, providing an intuitive way to effortlessly navigate and control VR and HCI applications. Considering increased privacy requirements, radar sensors emerge as a compelling alternative to cam…
▽ More
In the fast-paced field of human-computer interaction (HCI) and virtual reality (VR), automatic gesture recognition has become increasingly essential. This is particularly true for the recognition of hand signs, providing an intuitive way to effortlessly navigate and control VR and HCI applications. Considering increased privacy requirements, radar sensors emerge as a compelling alternative to cameras. They operate effectively in low-light conditions without capturing identifiable human details, thanks to their lower resolution and distinct wavelength compared to visible light.
While previous works predominantly deploy radar sensors for dynamic hand gesture recognition based on Doppler information, our approach prioritizes classification using an imaging radar that operates on spatial information, e.g. image-like data. However, generating large training datasets required for neural networks (NN) is a time-consuming and challenging process, often falling short of covering all potential scenarios. Acknowledging these challenges, this study explores the efficacy of synthetic data generated by an advanced radar ray-tracing simulator. This simulator employs an intuitive material model that can be adjusted to introduce data diversity.
Despite exclusively training the NN on synthetic data, it demonstrates promising performance when put to the test with real measurement data. This emphasizes the practicality of our methodology in overcoming data scarcity challenges and advancing the field of automatic gesture recognition in VR and HCI applications.
△ Less
Submitted 20 February, 2024;
originally announced February 2024.
-
UAV Formation Optimization for Communication-assisted InSAR Sensing
Authors:
Mohamed-Amine Lahmeri,
Victor Mustieles-Pérez,
Martin Vossiek,
Gerhard Krieger,
Robert Schober
Abstract:
Interferometric synthetic aperture radar (InSAR) is an increasingly important remote sensing technique that enables three-dimensional (3D) sensing applications such as the generation of accurate digital elevation models (DEMs). In this paper, we investigate the joint formation and communication resource allocation optimization for a system comprising two unmanned aerial vehicles (UAVs) to perform…
▽ More
Interferometric synthetic aperture radar (InSAR) is an increasingly important remote sensing technique that enables three-dimensional (3D) sensing applications such as the generation of accurate digital elevation models (DEMs). In this paper, we investigate the joint formation and communication resource allocation optimization for a system comprising two unmanned aerial vehicles (UAVs) to perform InSAR sensing and to transfer the acquired data to the ground. To this end, we adopt as sensing performance metrics the interferometric coherence, i.e., the local correlation between the two co-registered UAV radar images, and the height of ambiguity (HoA), which together are a measure for the accuracy with which the InSAR system can estimate the height of ground objects. In addition, an analytical expression for the coverage of the considered InSAR sensing system is derived. Our objective is to maximize the InSAR coverage while satisfying all relevant InSAR-specific sensing and communication performance metrics. To tackle the non-convexity of the formulated optimization problem, we employ alternating optimization (AO) techniques combined with successive convex approximation (SCA). Our simulation results reveal that the resulting resource allocation algorithm outperforms two benchmark schemes in terms of InSAR coverage while satisfying all sensing and real-time communication requirements. Furthermore, we highlight the importance of efficient communication resource allocation in facilitating real-time sensing and unveil the trade-off between InSAR height estimation accuracy and coverage.
△ Less
Submitted 12 November, 2023;
originally announced November 2023.
-
Joint Transmit Signal and Beamforming Design for Integrated Sensing and Power Transfer Systems
Authors:
Kenneth MacSporran Mayer,
Nikita Shanin,
Zhenlong You,
Sebastian Lotter,
Stefan Brückner,
Martin Vossiek,
Laura Cottatellucci,
Robert Schober
Abstract:
Integrating different functionalities, conventionally implemented as dedicated systems, into a single platform allows utilising the available resources more efficiently. We consider an integrated sensing and power transfer (ISAPT) system and propose the joint optimisation of the rectangular pulse-shaped transmit signal and the beamforming vector to combine sensing and wireless power transfer (WPT)…
▽ More
Integrating different functionalities, conventionally implemented as dedicated systems, into a single platform allows utilising the available resources more efficiently. We consider an integrated sensing and power transfer (ISAPT) system and propose the joint optimisation of the rectangular pulse-shaped transmit signal and the beamforming vector to combine sensing and wireless power transfer (WPT) functionalities efficiently. In contrast to prior works, we adopt an accurate non-linear circuit-based energy harvesting (EH) model. We formulate and solve a non-convex optimisation problem for a general number of EH receivers to maximise a weighted sum of the average harvested powers at the EH receivers while ensuring the received echo signal reflected by a sensing target (ST) has sufficient power for estimating the range to the ST with a prescribed accuracy within the considered coverage region. The average harvested power is shown to monotonically increase with the pulse duration when the average transmit power budget is sufficiently large. We discuss the trade-off between sensing performance and power transfer for the considered ISAPT system. The proposed approach significantly outperforms a heuristic baseline scheme based on a linear EH model, which linearly combines energy beamforming with the beamsteering vector in the direction to the ST as its transmit strategy.
△ Less
Submitted 20 January, 2024; v1 submitted 8 November, 2023;
originally announced November 2023.
-
Concept for an Automatic Annotation of Automotive Radar Data Using AI-segmented Aerial Camera Images
Authors:
Marcel Hoffmann,
Sandro Braun,
Oliver Sura,
Michael Stelzig,
Christian Schüßler,
Knut Graichen,
Martin Vossiek
Abstract:
This paper presents an approach to automatically annotate automotive radar data with AI-segmented aerial camera images. For this, the images of an unmanned aerial vehicle (UAV) above a radar vehicle are panoptically segmented and mapped in the ground plane onto the radar images. The detected instances and segments in the camera image can then be applied directly as labels for the radar data. Owing…
▽ More
This paper presents an approach to automatically annotate automotive radar data with AI-segmented aerial camera images. For this, the images of an unmanned aerial vehicle (UAV) above a radar vehicle are panoptically segmented and mapped in the ground plane onto the radar images. The detected instances and segments in the camera image can then be applied directly as labels for the radar data. Owing to the advantageous bird's eye position, the UAV camera does not suffer from optical occlusion and is capable of creating annotations within the complete field of view of the radar. The effectiveness and scalability are demonstrated in measurements, where 589 pedestrians in the radar data were automatically labeled within 2 minutes.
△ Less
Submitted 1 September, 2023;
originally announced September 2023.
-
A Realistic Radar Ray Tracing Simulator for Hand Pose Imaging
Authors:
Johanna Bräunig,
Christian Schüßler,
Vanessa Wirth,
Marc Stamminger,
Ingrid Ullmann,
Martin Vossiek
Abstract:
With the increasing popularity of human-computer interaction applications, there is also growing interest in generating sufficiently large and diverse data sets for automatic radar-based recognition of hand poses and gestures. Radar simulations are a vital approach to generating training data (e.g., for machine learning). Therefore, this work applies a ray tracing method to radar imaging of the ha…
▽ More
With the increasing popularity of human-computer interaction applications, there is also growing interest in generating sufficiently large and diverse data sets for automatic radar-based recognition of hand poses and gestures. Radar simulations are a vital approach to generating training data (e.g., for machine learning). Therefore, this work applies a ray tracing method to radar imaging of the hand. The performance of the proposed simulation approach is verified by a comparison of simulation and measurement data based on an imaging radar with a high lateral resolution. In addition, the surface material model incorporated into the ray tracer is highlighted in more detail and parameterized for radar hand imaging. Measurements and simulations show a very high similarity between synthetic and real radar image captures. The presented results demonstrate that it is possible to generate very realistic simulations of radar measurement data even for complex radar hand pose imaging systems.
△ Less
Submitted 28 July, 2023;
originally announced July 2023.
-
Super-Resolution Radar Imaging with Sparse Arrays Using a Deep Neural Network Trained with Enhanced Virtual Data
Authors:
Christian Schuessler,
Marcel Hoffmann,
Martin Vossiek
Abstract:
This paper introduces a method based on a deep neural network (DNN) that is perfectly capable of processing radar data from extremely thinned radar apertures. The proposed DNN processing can provide both aliasing-free radar imaging and super-resolution. The results are validated by measuring the detection performance on realistic simulation data and by evaluating the Point-Spread-function (PSF) an…
▽ More
This paper introduces a method based on a deep neural network (DNN) that is perfectly capable of processing radar data from extremely thinned radar apertures. The proposed DNN processing can provide both aliasing-free radar imaging and super-resolution. The results are validated by measuring the detection performance on realistic simulation data and by evaluating the Point-Spread-function (PSF) and the target-separation performance on measured point-like targets. Also, a qualitative evaluation of a typical automotive scene is conducted. It is shown that this approach can outperform state-of-the-art subspace algorithms and also other existing machine learning solutions. The presented results suggest that machine learning approaches trained with sufficiently sophisticated virtual input data are a very promising alternative to compressed sensing and subspace approaches in radar signal processing. The key to this performance is that the DNN is trained using realistic simulation data that perfectly mimic a given sparse antenna radar array hardware as the input. As ground truth, ultra-high resolution data from an enhanced virtual radar are simulated. Contrary to other work, the DNN utilizes the complete radar cube and not only the antenna channel information at certain range-Doppler detections. After training, the proposed DNN is capable of sidelobe- and ambiguity-free imaging. It simultaneously delivers nearly the same resolution and image quality as would be achieved with a fully occupied array.
△ Less
Submitted 16 June, 2023;
originally announced June 2023.
-
Implementation of Real-Time Automotive SAR Imaging
Authors:
Marcel Hoffmann,
Theresa Noegel,
Christian Schüßler,
Lars Schwenger,
Peter Gulden,
Dietmar Fey,
Martin Vossiek
Abstract:
This paper presents measures to reduce the computation time of automotive synthetic aperture radar (SAR) imaging to achieve real-time capability. For this, the image formation, which is based on the Back-Projection algorithm, was thoroughly analyzed. Various optimizations were individually tested and analyzed on graphics processing units (GPU). Apart from the time reduction gained from these measu…
▽ More
This paper presents measures to reduce the computation time of automotive synthetic aperture radar (SAR) imaging to achieve real-time capability. For this, the image formation, which is based on the Back-Projection algorithm, was thoroughly analyzed. Various optimizations were individually tested and analyzed on graphics processing units (GPU). Apart from the time reduction gained from these measures, the data size needed for processing was also drastically decreased. With a combination of all measures, a high-resolution SAR image of 30 m by 30 m that combines 8192 chirps can be reconstructed in less than 30 ms using a standard GPU. It is thus demonstrated that a real-time implementation of automotive SAR is possible.
△ Less
Submitted 16 June, 2023;
originally announced June 2023.
-
Achieving Efficient and Realistic Full-Radar Simulations and Automatic Data Annotation by exploiting Ray Meta Data of a Radar Ray Tracing Simulator
Authors:
Christian Schüßler,
Marcel Hoffmann,
Vanessa Wirth,
Björn Eskofier,
Tim Weyrich,
Marc Stamminger,
Martin Vossiek
Abstract:
In this work a novel radar simulation concept is introduced that allows to simulate realistic radar data for Range, Doppler, and for arbitrary antenna positions in an efficient way. Further, it makes it possible to automatically annotate the simulated radar signal by allowing to decompose it into different parts. This approach allows not only almost perfect annotations possible, but also allows th…
▽ More
In this work a novel radar simulation concept is introduced that allows to simulate realistic radar data for Range, Doppler, and for arbitrary antenna positions in an efficient way. Further, it makes it possible to automatically annotate the simulated radar signal by allowing to decompose it into different parts. This approach allows not only almost perfect annotations possible, but also allows the annotation of exotic effects, such as multi-path effects or to label signal parts originating from different parts of an object. This is possible by adapting the computation process of a Monte Carlo shooting and bouncing rays (SBR) simulator. By considering the hits of each simulated ray, various meta data can be stored such as hit position, mesh pointer, object IDs, and many more. This collected meta data can then be utilized to predict the change of path lengths introduced by object motion to obtain Doppler information or to apply specific ray filter rules in order obtain radar signals that only fulfil specific conditions, such as multiple bounces or containing specific object IDs. Using this approach, perfect and otherwise almost impossible annotations schemes can be realized.
△ Less
Submitted 23 May, 2023;
originally announced May 2023.
-
ShaRPy: Shape Reconstruction and Hand Pose Estimation from RGB-D with Uncertainty
Authors:
Vanessa Wirth,
Anna-Maria Liphardt,
Birte Coppers,
Johanna Bräunig,
Simon Heinrich,
Sigrid Leyendecker,
Arnd Kleyer,
Georg Schett,
Martin Vossiek,
Bernhard Egger,
Marc Stamminger
Abstract:
Despite their potential, markerless hand tracking technologies are not yet applied in practice to the diagnosis or monitoring of the activity in inflammatory musculoskeletal diseases. One reason is that the focus of most methods lies in the reconstruction of coarse, plausible poses, whereas in the clinical context, accurate, interpretable, and reliable results are required. Therefore, we propose S…
▽ More
Despite their potential, markerless hand tracking technologies are not yet applied in practice to the diagnosis or monitoring of the activity in inflammatory musculoskeletal diseases. One reason is that the focus of most methods lies in the reconstruction of coarse, plausible poses, whereas in the clinical context, accurate, interpretable, and reliable results are required. Therefore, we propose ShaRPy, the first RGB-D Shape Reconstruction and hand Pose tracking system, which provides uncertainty estimates of the computed pose, e.g., when a finger is hidden or its estimate is inconsistent with the observations in the input, to guide clinical decision-making. Besides pose, ShaRPy approximates a personalized hand shape, promoting a more realistic and intuitive understanding of its digital twin. Our method requires only a light-weight setup with a single consumer-level RGB-D camera yet it is able to distinguish similar poses with only small joint angle deviations in a metrically accurate space. This is achieved by combining a data-driven dense correspondence predictor with traditional energy minimization. To bridge the gap between interactive visualization and biomedical simulation we leverage a parametric hand model in which we incorporate biomedical constraints and optimize for both, its pose and hand shape. We evaluate ShaRPy on a keypoint detection benchmark and show qualitative results of hand function assessments for activity monitoring of musculoskeletal diseases.
△ Less
Submitted 12 September, 2023; v1 submitted 17 March, 2023;
originally announced March 2023.
