-
Lidar Upsampling with Sliced Wasserstein Distance
Authors:
Artem Savkin,
Yida Wang,
Sebastian Wirkert,
Nassir Navab,
Federico Tombar
Abstract:
Lidar became an important component of the perception systems in autonomous driving. But challenges of training data acquisition and annotation made emphasized the role of the sensor to sensor domain adaptation. In this work, we address the problem of lidar upsampling. Learning on lidar point clouds is rather a challenging task due to their irregular and sparse structure. Here we propose a method…
▽ More
Lidar became an important component of the perception systems in autonomous driving. But challenges of training data acquisition and annotation made emphasized the role of the sensor to sensor domain adaptation. In this work, we address the problem of lidar upsampling. Learning on lidar point clouds is rather a challenging task due to their irregular and sparse structure. Here we propose a method for lidar point cloud upsampling which can reconstruct fine-grained lidar scan patterns. The key idea is to utilize edge-aware dense convolutions for both feature extraction and feature expansion. Additionally applying a more accurate Sliced Wasserstein Distance facilitates learning of the fine lidar sweep structures. This in turn enables our method to employ a one-stage upsampling paradigm without the need for coarse and fine reconstruction. We conduct several experiments to evaluate our method and demonstrate that it provides better upsampling.
△ Less
Submitted 31 January, 2023;
originally announced January 2023.
-
Video-rate multispectral imaging in laparoscopic surgery: First-in-human application
Authors:
Leonardo Ayala,
Sebastian Wirkert,
Anant Vemuri,
Tim Adler,
Silvia Seidlitz,
Sebastian Pirmann,
Christina Engels,
Dogu Teber,
Lena Maier-Hein
Abstract:
Multispectral and hyperspectral imaging (MSI/HSI) can provide clinically relevant information on morphological and functional tissue properties. Application in the operating room (OR), however, has so far been limited by complex hardware setups and slow acquisition times. To overcome these limitations, we propose a novel imaging system for video-rate spectral imaging in the clinical workflow. The…
▽ More
Multispectral and hyperspectral imaging (MSI/HSI) can provide clinically relevant information on morphological and functional tissue properties. Application in the operating room (OR), however, has so far been limited by complex hardware setups and slow acquisition times. To overcome these limitations, we propose a novel imaging system for video-rate spectral imaging in the clinical workflow. The system integrates a small snapshot multispectral camera with a standard laparoscope and a clinically commonly used light source, enabling the recording of multispectral images with a spectral dimension of 16 at a frame rate of 25 Hz. An ongoing in patient study shows that multispectral recordings from this system can help detect perfusion changes in partial nephrectomy surgery, thus opening the doors to a wide range of clinical applications.
△ Less
Submitted 28 May, 2021;
originally announced May 2021.
-
Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
Authors:
Sebastian Houben,
Stephanie Abrecht,
Maram Akila,
Andreas Bär,
Felix Brockherde,
Patrick Feifel,
Tim Fingscheidt,
Sujan Sai Gannamaneni,
Seyed Eghbal Ghobadi,
Ahmed Hammam,
Anselm Haselhoff,
Felix Hauser,
Christian Heinzemann,
Marco Hoffmann,
Nikhil Kapoor,
Falk Kappel,
Marvin Klingner,
Jan Kronenberger,
Fabian Küppers,
Jonas Löhdefink,
Michael Mlynarski,
Michael Mock,
Firas Mualla,
Svetlana Pavlitskaya,
Maximilian Poretschkin
, et al. (16 additional authors not shown)
Abstract:
The use of deep neural networks (DNNs) in safety-critical applications like mobile health and autonomous driving is challenging due to numerous model-inherent shortcomings. These shortcomings are diverse and range from a lack of generalization over insufficient interpretability to problems with malicious inputs. Cyber-physical systems employing DNNs are therefore likely to suffer from safety conce…
▽ More
The use of deep neural networks (DNNs) in safety-critical applications like mobile health and autonomous driving is challenging due to numerous model-inherent shortcomings. These shortcomings are diverse and range from a lack of generalization over insufficient interpretability to problems with malicious inputs. Cyber-physical systems employing DNNs are therefore likely to suffer from safety concerns. In recent years, a zoo of state-of-the-art techniques aiming to address these safety concerns has emerged. This work provides a structured and broad overview of them. We first identify categories of insufficiencies to then describe research activities aiming at their detection, quantification, or mitigation. Our paper addresses both machine learning experts and safety engineers: The former ones might profit from the broad range of machine learning topics covered and discussions on limitations of recent methods. The latter ones might gain insights into the specifics of modern ML methods. We moreover hope that our contribution fuels discussions on desiderata for ML systems and strategies on how to propel existing approaches accordingly.
△ Less
Submitted 29 April, 2021;
originally announced April 2021.
-
Band selection for oxygenation estimation with multispectral/hyperspectral imaging
Authors:
Leonardo A. Ayala,
Fabian Isensee,
Sebastian J. Wirkert,
Anant S. Vemuri,
Klaus H. Maier-Hein,
Baowei Fei,
Lena Maier-Hein
Abstract:
Multispectral imaging provides valuable information on tissue composition such as hemoglobin oxygen saturation. However, the real-time application of this technique in interventional medicine can be challenging due to the long acquisition times needed for large amounts of hyperspectral data with hundreds of bands. While this challenge can partially be addressed by choosing a discriminative subset…
▽ More
Multispectral imaging provides valuable information on tissue composition such as hemoglobin oxygen saturation. However, the real-time application of this technique in interventional medicine can be challenging due to the long acquisition times needed for large amounts of hyperspectral data with hundreds of bands. While this challenge can partially be addressed by choosing a discriminative subset of bands, the band selection methods proposed to date are mainly restricted by the availability of often hard to obtain reference measurements. We address this bottleneck with a new approach to band selection that leverages highly accurate Monte Carlo (MC) simulations. We hypothesize that a so chosen small subset of bands can reproduce or even improve upon the results of a quasi continuous spectral measurement. We further investigate whether novel domain adaptation techniques can address the inevitable domain shift stemming from the use of simulations. Initial results based on in silico and in vivo experiments suggest that 10-20 bands are sufficient to closely reproduce results from spectral measurements with 101 bands in the 500-700 nm range. The investigated domain adaptation technique, which only requires unlabeled in vivo measurements, yielded better results than the pure in silico band selection method. Overall, our method could guide development of fast multispectral imaging systems suited for interventional use without relying on complex hardware setups or manually labeled data
△ Less
Submitted 20 August, 2021; v1 submitted 27 May, 2019;
originally announced May 2019.
-
Hyperspectral Camera Selection for Interventional Health-care
Authors:
Anant S. Vemuri,
Sebastian Wirkert,
Lena Maier-Hein
Abstract:
Hyperspectral imaging (HSI) is an emerging modality in health-care applications for disease diagnosis, tissue assessment and image-guided surgery. Tissue reflectances captured by a HSI camera encode physiological properties including oxygenation and blood volume fraction. Optimal camera properties such as filter responses depend crucially on the application, and choosing a suitable HSI camera for…
▽ More
Hyperspectral imaging (HSI) is an emerging modality in health-care applications for disease diagnosis, tissue assessment and image-guided surgery. Tissue reflectances captured by a HSI camera encode physiological properties including oxygenation and blood volume fraction. Optimal camera properties such as filter responses depend crucially on the application, and choosing a suitable HSI camera for a research project and/or a clinical problem is not straightforward. We propose a generic framework for quantitative and application-specific performance assessment of HSI cameras and optical subsystem without the need for any physical setup. Based on user input about the camera characteristics and properties of the target domain, our framework quantifies the performance of the given camera configuration using large amounts of simulated data and a user-defined metric. The application of the framework to commercial camera selection and band selection in the context of oxygenation monitoring in interventional health-care demonstrates its integration into the design work-flow of an engineer. The advantage of being able to test the desired configuration without the need for purchasing expensive components may save system engineers valuable resources.
△ Less
Submitted 4 April, 2019;
originally announced April 2019.
-
Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks
Authors:
Tim J. Adler,
Lynton Ardizzone,
Anant Vemuri,
Leonardo Ayala,
Janek Gröhl,
Thomas Kirchner,
Sebastian Wirkert,
Jakob Kruse,
Carsten Rother,
Ullrich Köthe,
Lena Maier-Hein
Abstract:
Purpose: Optical imaging is evolving as a key technique for advanced sensing in the operating room. Recent research has shown that machine learning algorithms can be used to address the inverse problem of converting pixel-wise multispectral reflectance measurements to underlying tissue parameters, such as oxygenation. Assessment of the specific hardware used in conjunction with such algorithms, ho…
▽ More
Purpose: Optical imaging is evolving as a key technique for advanced sensing in the operating room. Recent research has shown that machine learning algorithms can be used to address the inverse problem of converting pixel-wise multispectral reflectance measurements to underlying tissue parameters, such as oxygenation. Assessment of the specific hardware used in conjunction with such algorithms, however, has not properly addressed the possibility that the problem may be ill-posed.
Methods: We present a novel approach to the assessment of optical imaging modalities, which is sensitive to the different types of uncertainties that may occur when inferring tissue parameters. Based on the concept of invertible neural networks, our framework goes beyond point estimates and maps each multispectral measurement to a full posterior probability distribution which is capable of representing ambiguity in the solution via multiple modes. Performance metrics for a hardware setup can then be computed from the characteristics of the posteriors.
Results: Application of the assessment framework to the specific use case of camera selection for physiological parameter estimation yields the following insights: (1) Estimation of tissue oxygenation from multispectral images is a well-posed problem, while (2) blood volume fraction may not be recovered without ambiguity. (3) In general, ambiguity may be reduced by increasing the number of spectral bands in the camera.
Conclusion: Our method could help to optimize optical camera design in an application-specific manner.
△ Less
Submitted 8 March, 2019;
originally announced March 2019.
-
nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation
Authors:
Fabian Isensee,
Jens Petersen,
Andre Klein,
David Zimmerer,
Paul F. Jaeger,
Simon Kohl,
Jakob Wasserthal,
Gregor Koehler,
Tobias Norajitra,
Sebastian Wirkert,
Klaus H. Maier-Hein
Abstract:
The U-Net was presented in 2015. With its straight-forward and successful architecture it quickly evolved to a commonly used benchmark in medical image segmentation. The adaptation of the U-Net to novel problems, however, comprises several degrees of freedom regarding the exact architecture, preprocessing, training and inference. These choices are not independent of each other and substantially im…
▽ More
The U-Net was presented in 2015. With its straight-forward and successful architecture it quickly evolved to a commonly used benchmark in medical image segmentation. The adaptation of the U-Net to novel problems, however, comprises several degrees of freedom regarding the exact architecture, preprocessing, training and inference. These choices are not independent of each other and substantially impact the overall performance. The present paper introduces the nnU-Net ('no-new-Net'), which refers to a robust and self-adapting framework on the basis of 2D and 3D vanilla U-Nets. We argue the strong case for taking away superfluous bells and whistles of many proposed network designs and instead focus on the remaining aspects that make out the performance and generalizability of a method. We evaluate the nnU-Net in the context of the Medical Segmentation Decathlon challenge, which measures segmentation performance in ten disciplines comprising distinct entities, image modalities, image geometries and dataset sizes, with no manual adjustments between datasets allowed. At the time of manuscript submission, nnU-Net achieves the highest mean dice scores across all classes and seven phase 1 tasks (except class 1 in BrainTumour) in the online leaderboard of the challenge.
△ Less
Submitted 27 September, 2018;
originally announced September 2018.
-
Analyzing Inverse Problems with Invertible Neural Networks
Authors:
Lynton Ardizzone,
Jakob Kruse,
Sebastian Wirkert,
Daniel Rahner,
Eric W. Pellegrini,
Ralf S. Klessen,
Lena Maier-Hein,
Carsten Rother,
Ullrich Köthe
Abstract:
In many tasks, in particular in natural science, the goal is to determine hidden system parameters from a set of measurements. Often, the forward process from parameter- to measurement-space is a well-defined function, whereas the inverse problem is ambiguous: one measurement may map to multiple different sets of parameters. In this setting, the posterior parameter distribution, conditioned on an…
▽ More
In many tasks, in particular in natural science, the goal is to determine hidden system parameters from a set of measurements. Often, the forward process from parameter- to measurement-space is a well-defined function, whereas the inverse problem is ambiguous: one measurement may map to multiple different sets of parameters. In this setting, the posterior parameter distribution, conditioned on an input measurement, has to be determined. We argue that a particular class of neural networks is well suited for this task -- so-called Invertible Neural Networks (INNs). Although INNs are not new, they have, so far, received little attention in literature. While classical neural networks attempt to solve the ambiguous inverse problem directly, INNs are able to learn it jointly with the well-defined forward process, using additional latent output variables to capture the information otherwise lost. Given a specific measurement and sampled latent variables, the inverse pass of the INN provides a full distribution over parameter space. We verify experimentally, on artificial data and real-world problems from astrophysics and medicine, that INNs are a powerful analysis tool to find multi-modalities in parameter space, to uncover parameter correlations, and to identify unrecoverable parameters.
△ Less
Submitted 6 February, 2019; v1 submitted 14 August, 2018;
originally announced August 2018.
-
Uncertainty-Aware Organ Classification for Surgical Data Science Applications in Laparoscopy
Authors:
S. Moccia,
S. J. Wirkert,
H. Kenngott,
A. S. Vemuri,
M. Apitz,
B. Mayer,
E. De Momi,
L. S. Mattos,
L. Maier-Hein
Abstract:
Objective: Surgical data science is evolving into a research field that aims to observe everything occurring within and around the treatment process to provide situation-aware data-driven assistance. In the context of endoscopic video analysis, the accurate classification of organs in the field of view of the camera proffers a technical challenge. Herein, we propose a new approach to anatomical st…
▽ More
Objective: Surgical data science is evolving into a research field that aims to observe everything occurring within and around the treatment process to provide situation-aware data-driven assistance. In the context of endoscopic video analysis, the accurate classification of organs in the field of view of the camera proffers a technical challenge. Herein, we propose a new approach to anatomical structure classification and image tagging that features an intrinsic measure of confidence to estimate its own performance with high reliability and which can be applied to both RGB and multispectral imaging (MI) data. Methods: Organ recognition is performed using a superpixel classification strategy based on textural and reflectance information. Classification confidence is estimated by analyzing the dispersion of class probabilities. Assessment of the proposed technology is performed through a comprehensive in vivo study with seven pigs. Results: When applied to image tagging, mean accuracy in our experiments increased from 65% (RGB) and 80% (MI) to 90% (RGB) and 96% (MI) with the confidence measure. Conclusion: Results showed that the confidence measure had a significant influence on the classification accuracy, and MI data are better suited for anatomical structure labeling than RGB data. Significance: This work significantly enhances the state of art in automatic labeling of endoscopic videos by introducing the use of the confidence metric, and by being the first study to use MI data for in vivo laparoscopic tissue classification. The data of our experiments will be released as the first in vivo MI dataset upon publication of this paper.
△ Less
Submitted 19 October, 2018; v1 submitted 21 June, 2017;
originally announced June 2017.
-
Path Assignment Techniques For Vehicle Tracking
Authors:
Richard Altendorfer,
Sebastian Wirkert
Abstract:
Many driver assistance systems such as Adaptive Cruise Control require the identification of the closest vehicle that is in the host vehicle's path. This entails an assignment of detected vehicles to the host vehicle path or neighboring paths. After reviewing approaches to the estimation of the host vehicle path and lane assignment techniques we introduce two methods that are motivated by the rati…
▽ More
Many driver assistance systems such as Adaptive Cruise Control require the identification of the closest vehicle that is in the host vehicle's path. This entails an assignment of detected vehicles to the host vehicle path or neighboring paths. After reviewing approaches to the estimation of the host vehicle path and lane assignment techniques we introduce two methods that are motivated by the rationale to filter measured data as late in the processing stages as possible in order to avoid delays and other artifacts of intermediate filters. These filters generate discrete posterior probability distributions from which a path or "lane" index is extracted by a median estimator. The relative performance of those methods is illustrated by a ROC using experimental data and labeled ground truth data.
△ Less
Submitted 11 February, 2017;
originally announced February 2017.
-
A Complete Derivation Of The Association Log-Likelihood Distance For Multi-Object Tracking
Authors:
Richard Altendorfer,
Sebastian Wirkert
Abstract:
The Mahalanobis distance is commonly used in multi-object trackers for measurement-to-track association. Starting with the original definition of the Mahalanobis distance we review its use in association. Given that there is no principle in multi-object tracking that sets the Mahalanobis distance apart as a distinguished statistical distance we revisit the global association hypotheses of multiple…
▽ More
The Mahalanobis distance is commonly used in multi-object trackers for measurement-to-track association. Starting with the original definition of the Mahalanobis distance we review its use in association. Given that there is no principle in multi-object tracking that sets the Mahalanobis distance apart as a distinguished statistical distance we revisit the global association hypotheses of multiple hypothesis tracking as the most general association setting. Those association hypotheses induce a distance-like quantity for assignment which we refer to as association log-likelihood distance. We compare the ability of the Mahalanobis distance to the association log-likelihood distance to yield correct association relations in Monte-Carlo simulations. It turns out that on average the distance based on association log-likelihood performs better than the Mahalanobis distance, confirming that the maximization of global association hypotheses is a more fundamental approach to association than the minimization of a certain statistical distance measure.
△ Less
Submitted 8 September, 2015; v1 submitted 17 August, 2015;
originally announced August 2015.
