Showing 1–15 of 15 results for author: Willomitzer, F

3D Imaging of Complex Specular Surfaces by Fusing Polarimetric and Deflectometric Information

Authors: Jiazhang Wang, Oliver Cossairt, Florian Willomitzer

Abstract: Accurate and fast 3D imaging of specular surfaces still poses major challenges for state-of-the-art optical measurement principles. Frequently used methods, such as phase-measuring deflectometry (PMD) or shape-from-polarization (SfP), rely on strong assumptions about the measured objects, limiting their generalizability in broader application areas like medical imaging, industrial inspection, virt… ▽ More Accurate and fast 3D imaging of specular surfaces still poses major challenges for state-of-the-art optical measurement principles. Frequently used methods, such as phase-measuring deflectometry (PMD) or shape-from-polarization (SfP), rely on strong assumptions about the measured objects, limiting their generalizability in broader application areas like medical imaging, industrial inspection, virtual reality, or cultural heritage analysis. In this paper, we introduce a measurement principle that utilizes a novel technique to effectively encode and decode the information contained in a light field reflected off a specular surface. We combine polarization cues from SfP with geometric information obtained from PMD to resolve all arising ambiguities in the 3D measurement. Moreover, our approach removes the unrealistic orthographic imaging assumption for SfP, which significantly improves the respective results. We showcase our new technique by demonstrating single-shot and multi-shot measurements on complex-shaped specular surfaces, displaying an evaluated accuracy of surface normals below $0.6^\circ$. △ Less

Submitted 4 June, 2024; originally announced June 2024.

Event-based Motion-Robust Accurate Shape Estimation for Mixed Reflectance Scenes

Authors: Aniket Dashpute, Jiazhang Wang, James Taylor, Oliver Cossairt, Ashok Veeraraghavan, Florian Willomitzer

Abstract: Event-based structured light systems have recently been introduced as an exciting alternative to conventional frame-based triangulation systems for the 3D measurements of diffuse surfaces. Important benefits include the fast capture speed and the high dynamic range provided by the event camera - albeit at the cost of lower data quality. So far, both low-accuracy event-based and high-accuracy frame… ▽ More Event-based structured light systems have recently been introduced as an exciting alternative to conventional frame-based triangulation systems for the 3D measurements of diffuse surfaces. Important benefits include the fast capture speed and the high dynamic range provided by the event camera - albeit at the cost of lower data quality. So far, both low-accuracy event-based and high-accuracy frame-based 3D imaging systems are tailored to a specific surface type, such as diffuse or specular, and can not be used for a broader class of object surfaces ("mixed reflectance scenes"). In this work, we present a novel event-based structured light system that enables fast 3D imaging of mixed reflectance scenes with high accuracy. On the captured events, we use epipolar constraints that intrinsically enable decomposing the measured reflections into diffuse, two-bounce specular, and other multi-bounce reflections. The diffuse surfaces in the scene are reconstructed using triangulation. Then, the reconstructed diffuse scene parts are leveraged as a "display" to evaluate the specular scene parts via deflectometry. This novel procedure allows us to use the entire scene as a virtual screen, using only a scanning laser and an event camera. The resulting system achieves fast and motion-robust (14Hz) reconstructions of mixed reflectance scenes with < 600 $μm$ depth error. Moreover, we introduce an "ultrafast" capture mode (250Hz) for the 3D measurement of diffuse scenes. △ Less

Submitted 10 June, 2025; v1 submitted 16 November, 2023; originally announced November 2023.

Accurate Eye Tracking from Dense 3D Surface Reconstructions using Single-Shot Deflectometry

Authors: Jiazhang Wang, Tianfu Wang, Bingjie Xu, Oliver Cossairt, Florian Willomitzer

Abstract: Eye-tracking plays a crucial role in the development of virtual reality devices, neuroscience research, and psychology. Despite its significance in numerous applications, achieving an accurate, robust, and fast eye-tracking solution remains a considerable challenge for current state-of-the-art methods. While existing reflection-based techniques (e.g., "glint tracking") are considered to be very ac… ▽ More Eye-tracking plays a crucial role in the development of virtual reality devices, neuroscience research, and psychology. Despite its significance in numerous applications, achieving an accurate, robust, and fast eye-tracking solution remains a considerable challenge for current state-of-the-art methods. While existing reflection-based techniques (e.g., "glint tracking") are considered to be very accurate, their performance is limited by their reliance on sparse 3D surface data acquired solely from the cornea surface. In this paper, we rethink the way how specular reflections can be used for eye tracking: We propose a novel method for accurate and fast evaluation of the gaze direction that exploits teachings from single-shot phase-measuring-deflectometry(PMD). In contrast to state-of-the-art reflection-based methods, our method acquires dense 3D surface information of both cornea and sclera within only one single camera frame (single-shot). For a typical measurement, we acquire $>3000 \times$ more surface reflection points ("glints") than conventional methods. We show the feasibility of our approach with experimentally evaluated gaze errors on a realistic model eye below only $0.12^\circ$. Moreover, we demonstrate quantitative measurements on real human eyes in vivo, reaching accuracy values between only $0.46^\circ$ and $0.97^\circ$. △ Less

Submitted 19 November, 2024; v1 submitted 14 August, 2023; originally announced August 2023.

Accurate and Fast VR Eye-Tracking using Deflectometric Information

Authors: Jiazhang Wang, Tianfu Wang, Bingjie Xu, Oliver Cossairt, Florian Willomitzer

Abstract: We present two methods for fast and precise eye-tracking in VR headsets. Both methods exploit deflectometric information, i.e., the specular reflection of an extended screen over the eye surface. We present two methods for fast and precise eye-tracking in VR headsets. Both methods exploit deflectometric information, i.e., the specular reflection of an extended screen over the eye surface. △ Less

Submitted 12 April, 2023; originally announced April 2023.

Thermal Spread Functions (TSF): Physics-guided Material Classification

Authors: Aniket Dashpute, Vishwanath Saragadam, Emma Alexander, Florian Willomitzer, Aggelos Katsaggelos, Ashok Veeraraghavan, Oliver Cossairt

Abstract: Robust and non-destructive material classification is a challenging but crucial first-step in numerous vision applications. We propose a physics-guided material classification framework that relies on thermal properties of the object. Our key observation is that the rate of heating and cooling of an object depends on the unique intrinsic properties of the material, namely the emissivity and diffus… ▽ More Robust and non-destructive material classification is a challenging but crucial first-step in numerous vision applications. We propose a physics-guided material classification framework that relies on thermal properties of the object. Our key observation is that the rate of heating and cooling of an object depends on the unique intrinsic properties of the material, namely the emissivity and diffusivity. We leverage this observation by gently heating the objects in the scene with a low-power laser for a fixed duration and then turning it off, while a thermal camera captures measurements during the heating and cooling process. We then take this spatial and temporal "thermal spread function" (TSF) to solve an inverse heat equation using the finite-differences approach, resulting in a spatially varying estimate of diffusivity and emissivity. These tuples are then used to train a classifier that produces a fine-grained material label at each spatial pixel. Our approach is extremely simple requiring only a small light source (low power laser) and a thermal camera, and produces robust classification results with 86% accuracy over 16 classes. △ Less

Submitted 2 April, 2023; originally announced April 2023.

Optimization-Based Eye Tracking using Deflectometric Information

Authors: Tianfu Wang, Jiazhang Wang, Oliver Cossairt, Florian Willomitzer

Abstract: Eye tracking is an important tool with a wide range of applications in Virtual, Augmented, and Mixed Reality (VR/AR/MR) technologies. State-of-the-art eye tracking methods are either reflection-based and track reflections of sparse point light sources, or image-based and exploit 2D features of the acquired eye image. In this work, we attempt to significantly improve reflection-based methods by uti… ▽ More Eye tracking is an important tool with a wide range of applications in Virtual, Augmented, and Mixed Reality (VR/AR/MR) technologies. State-of-the-art eye tracking methods are either reflection-based and track reflections of sparse point light sources, or image-based and exploit 2D features of the acquired eye image. In this work, we attempt to significantly improve reflection-based methods by utilizing pixel-dense deflectometric surface measurements in combination with optimization-based inverse rendering algorithms. Utilizing the known geometry of our deflectometric setup, we develop a differentiable rendering pipeline based on PyTorch3D that simulates a virtual eye under screen illumination. Eventually, we exploit the image-screen-correspondence information from the captured measurements to find the eye's rotation, translation, and shape parameters with our renderer via gradient descent. In general, our method does not require a specific pattern and can work with ordinary video frames of the main VR/AR/MR screen itself. We demonstrate real-world experiments with evaluated mean relative gaze errors below 0.45 degrees at a precision better than 0.11 degrees. Moreover, we show an improvement of 6X over a representative reflection-based state-of-the-art method in simulation. △ Less

Submitted 8 March, 2023; originally announced March 2023.

Single-shot ToF sensing with sub-mm precision using conventional CMOS sensors

Authors: Manuel Ballester, Heming Wang, Jiren Li, Oliver Cossairt, Florian Willomitzer

Abstract: We present a novel single-shot interferometric ToF camera targeted for precise 3D measurements of dynamic objects. The camera concept is based on Synthetic Wavelength Interferometry, a technique that allows retrieval of depth maps of objects with optically rough surfaces at submillimeter depth precision. In contrast to conventional ToF cameras, our device uses only off-the-shelf CCD/CMOS detectors… ▽ More We present a novel single-shot interferometric ToF camera targeted for precise 3D measurements of dynamic objects. The camera concept is based on Synthetic Wavelength Interferometry, a technique that allows retrieval of depth maps of objects with optically rough surfaces at submillimeter depth precision. In contrast to conventional ToF cameras, our device uses only off-the-shelf CCD/CMOS detectors and works at their native chip resolution (as of today, theoretically up to 20 Mp and beyond). Moreover, we can obtain a full 3D model of the object in single-shot, meaning that no temporal sequence of exposures or temporal illumination modulation (such as amplitude or frequency modulation) is necessary, which makes our camera robust against object motion. In this paper, we introduce the novel camera concept and show first measurements that demonstrate the capabilities of our system. We present 3D measurements of small (cm-sized) objects with > 2 Mp point cloud resolution (the resolution of our used detector) and up to sub-mm depth precision. We also report a "single-shot 3D video" acquisition and a first single-shot "Non-Line-of-Sight" measurement. Our technique has great potential for high-precision applications with dynamic object movement, e.g., in AR/VR, industrial inspection, medical imaging, and imaging through scattering media like fog or human tissue. △ Less

Submitted 1 December, 2022; originally announced December 2022.

Synthetic Wavelength Imaging -- Utilizing Spectral Correlations for High-Precision Time-of-Flight Sensing

Authors: Florian Willomitzer

Abstract: This book chapter describes how spectral correlations in scattered light fields can be utilized for high-precision time-of-flight sensing. The chapter should serve as a gentle introduction and is intended for computational imaging scientists and students new to the fascinating topic of synthetic wavelength imaging. Technical details (such as detector or light source specifications) will be largely… ▽ More This book chapter describes how spectral correlations in scattered light fields can be utilized for high-precision time-of-flight sensing. The chapter should serve as a gentle introduction and is intended for computational imaging scientists and students new to the fascinating topic of synthetic wavelength imaging. Technical details (such as detector or light source specifications) will be largely omitted. Instead, the similarities between different methods will be emphasized to "draw the bigger picture." △ Less

Submitted 11 September, 2022; originally announced September 2022.

Can Deep Learning Assist Automatic Identification of Layered Pigments From XRF Data?

Authors: Bingjie, Xu, Yunan Wu, Pengxiao Hao, Marc Vermeulen, Alicia McGeachy, Kate Smith, Katherine Eremin, Georgina Rayner, Giovanni Verri, Florian Willomitzer, Matthias Alfeld, Jack Tumblin, Aggelos Katsaggelos, Marc Walton

Abstract: X-ray fluorescence spectroscopy (XRF) plays an important role for elemental analysis in a wide range of scientific fields, especially in cultural heritage. XRF imaging, which uses a raster scan to acquire spectra across artworks, provides the opportunity for spatial analysis of pigment distributions based on their elemental composition. However, conventional XRF-based pigment identification relies… ▽ More X-ray fluorescence spectroscopy (XRF) plays an important role for elemental analysis in a wide range of scientific fields, especially in cultural heritage. XRF imaging, which uses a raster scan to acquire spectra across artworks, provides the opportunity for spatial analysis of pigment distributions based on their elemental composition. However, conventional XRF-based pigment identification relies on time-consuming elemental mapping by expert interpretations of measured spectra. To reduce the reliance on manual work, recent studies have applied machine learning techniques to cluster similar XRF spectra in data analysis and to identify the most likely pigments. Nevertheless, it is still challenging for automatic pigment identification strategies to directly tackle the complex structure of real paintings, e.g. pigment mixtures and layered pigments. In addition, pixel-wise pigment identification based on XRF imaging remains an obstacle due to the high noise level compared with averaged spectra. Therefore, we developed a deep-learning-based end-to-end pigment identification framework to fully automate the pigment identification process. In particular, it offers high sensitivity to the underlying pigments and to the pigments with a low concentration, therefore enabling satisfying results in mapping the pigments based on single-pixel XRF spectrum. As case studies, we applied our framework to lab-prepared mock-up paintings and two 19th-century paintings: Paul Gauguin's Poèmes Barbares (1896) that contains layered pigments with an underlying painting, and Paul Cezanne's The Bathers (1899-1904). The pigment identification results demonstrated that our model achieved comparable results to the analysis by elemental mapping, suggesting the generalizability and stability of our model. △ Less

Submitted 26 July, 2022; originally announced July 2022.

Comments: 11 pages, 10 figures

Characterizing the Immaterial. Noninvasive Imaging and Analysis of Stephen Benton's Hologram Engine no. 9

Authors: Marc Walton, Pengxiao Hao, Marc Vermeulen, Florian Willomitzer, Oliver Cossairt

Abstract: Invented in 1962, holography is a unique merging of art and technology. It persisted at the scientific cutting edge through the 1990s, when digital imaging emerged and supplanted film. Today, holography is experiencing new interest as analog holograms enter major museum collections as bona fide works of art. In this essay, we articulate our initial steps at Northwestern's Center for Scientific Stu… ▽ More Invented in 1962, holography is a unique merging of art and technology. It persisted at the scientific cutting edge through the 1990s, when digital imaging emerged and supplanted film. Today, holography is experiencing new interest as analog holograms enter major museum collections as bona fide works of art. In this essay, we articulate our initial steps at Northwestern's Center for Scientific Studies in the Arts to describe the technological challenges on the conservation of holograms, emphasizing their nature as an active material. A holographic image requires user interaction to be viewed, and the materials are delicate and prone to deterioration. Specifically, we outline our methods for creating digital preservation copies of holographic artworks by documenting the wavefront of propagating light. In so doing, we demonstrate why it remains challenging to faithfully capture their high spatial resolution, the full parallax, and deep depths of field without terabytes of data. In addition, we use noninvasive analytical techniques such as spectral imaging, X-ray fluorescence, and optical coherence tomography, to provide insights on hologram material properties. Through these studies we hope to address current concerns about the long term preservation of holograms while translating this artform into a digital format to entice new audiences. △ Less

Submitted 12 October, 2021; originally announced October 2021.

SkinScan: Low-Cost 3D-Scanning for Dermatologic Diagnosis and Documentation

Authors: Merlin A. Nau, Florian Schiffers, Yunhao Li, Bingjie Xu, Andreas Maier, Jack Tumblin, Marc Walton, Aggelos K. Katsaggelos, Florian Willomitzer, Oliver Cossairt

Abstract: The utilization of computational photography becomes increasingly essential in the medical field. Today, imaging techniques for dermatology range from two-dimensional (2D) color imagery with a mobile device to professional clinical imaging systems measuring additional detailed three-dimensional (3D) data. The latter are commonly expensive and not accessible to a broad audience. In this work, we pr… ▽ More The utilization of computational photography becomes increasingly essential in the medical field. Today, imaging techniques for dermatology range from two-dimensional (2D) color imagery with a mobile device to professional clinical imaging systems measuring additional detailed three-dimensional (3D) data. The latter are commonly expensive and not accessible to a broad audience. In this work, we propose a novel system and software framework that relies only on low-cost (and even mobile) commodity devices present in every household to measure detailed 3D information of the human skin with a 3D-gradient-illumination-based method. We believe that our system has great potential for early-stage diagnosis and monitoring of skin diseases, especially in vastly populated or underdeveloped areas. △ Less

Submitted 31 January, 2021; originally announced February 2021.

Comments: 5 pages, 4 Figures, Submitted at ICIP 2021

Uncalibrated Deflectometry with a Mobile Device on Extended Specular Surfaces

Authors: Florian Willomitzer, Chia-Kai Yeh, Vikas Gupta, William Spies, Florian Schiffers, Marc Walton, Oliver Cossairt

Abstract: We introduce a system and methods for the three-dimensional measurement of extended specular surfaces with high surface normal variations. Our system consists only of a mobile hand held device and exploits screen and front camera for Deflectometry-based surface measurements. We demonstrate high quality measurements without the need for an offline calibration procedure. In addition, we develop a mu… ▽ More We introduce a system and methods for the three-dimensional measurement of extended specular surfaces with high surface normal variations. Our system consists only of a mobile hand held device and exploits screen and front camera for Deflectometry-based surface measurements. We demonstrate high quality measurements without the need for an offline calibration procedure. In addition, we develop a multi-view technique to compensate for the small screen of a mobile device so that large surfaces can be densely reconstructed in their entirety. This work is a first step towards developing a self-calibrating Deflectometry procedure capable of taking 3D surface measurements of specular objects in the wild and accessible to users with little to no technical imaging experience. △ Less

Submitted 24 July, 2019; originally announced July 2019.

Seminar Innovation Management - Winter Term 2017

Authors: Gerd Häusler, Aleksandra Milczarek, Markus Schreiter, Thomas Kästner, Florian Willomitzer, Andreas Maier, Florian Schiffers, Stefan Steidl, Temitope Paul Onanuga, Mathias Unberath, Florian Dötzer, Maike Stöve, Jonas Hajek, Christian Heidorn, Felix Häußler, Tobias Geimer, Johannes Wendel

Abstract: This document contains the results obtained by the Innovation Management Seminar in winter term 2017. In total 11 ideas have been developed by the team. In the document all 11 ideas show improvements for future applications in ophthalmology. The 11 ideas are AR/VR Glasses with Medical Applications, Augmented Reality Eye Surgery, Game Diagnosis, Intelligent Adapting Glasses, MD Facebook, Medical Cr… ▽ More This document contains the results obtained by the Innovation Management Seminar in winter term 2017. In total 11 ideas have been developed by the team. In the document all 11 ideas show improvements for future applications in ophthalmology. The 11 ideas are AR/VR Glasses with Medical Applications, Augmented Reality Eye Surgery, Game Diagnosis, Intelligent Adapting Glasses, MD Facebook, Medical Crowd Segmentation, Personalized 3D Model of the Human Eye, Photoacoustic Contact Lens, Power Supply Smart Contact Lens, VR-Cornea and Head Mount for Fundus Imaging △ Less

Submitted 22 August, 2017; originally announced August 2017.

ACM Class: K.6.0

Hand-guided 3D surface acquisition by combining simple light sectioning with real-time algorithms

Authors: Oliver Arold, Svenja Ettl, Florian Willomitzer, Gerd Häusler

Abstract: Precise 3D measurements of rigid surfaces are desired in many fields of application like quality control or surgery. Often, views from all around the object have to be acquired for a full 3D description of the object surface. We present a sensor principle called "Flying Triangulation" which avoids an elaborate "stop-and-go" procedure. It combines a low-cost classical light-section sensor with an a… ▽ More Precise 3D measurements of rigid surfaces are desired in many fields of application like quality control or surgery. Often, views from all around the object have to be acquired for a full 3D description of the object surface. We present a sensor principle called "Flying Triangulation" which avoids an elaborate "stop-and-go" procedure. It combines a low-cost classical light-section sensor with an algorithmic pipeline. A hand-guided sensor captures a continuous movie of 3D views while being moved around the object. The views are automatically aligned and the acquired 3D model is displayed in real time. In contrast to most existing sensors no bandwidth is wasted for spatial or temporal encoding of the projected lines. Nor is an expensive color camera necessary for 3D acquisition. The achievable measurement uncertainty and lateral resolution of the generated 3D data is merely limited by physics. An alternating projection of vertical and horizontal lines guarantees the existence of corresponding points in successive 3D views. This enables a precise registration without surface interpolation. For registration, a variant of the iterative closest point algorithm - adapted to the specific nature of our 3D views - is introduced. Furthermore, data reduction and smoothing without losing lateral resolution as well as the acquisition and mapping of a color texture is presented. The precision and applicability of the sensor is demonstrated by simulation and measurement results. △ Less

Submitted 9 January, 2014; originally announced January 2014.

Comments: 19 pages, 22 figures

Flying Triangulation - towards the 3D movie camera

Authors: Florian Willomitzer, Svenja Ettl, Christian Faber, Gerd Häusler

Abstract: Flying Triangulation sensors enable a free-hand and motion-robust 3D data acquisition of complex shaped objects. The measurement principle is based on a multi-line light-sectioning approach and uses sophisticated algorithms for real-time registration (S. Ettl et al., Appl. Opt. 51 (2012) 281-289). As "single-shot principle", light sectioning enables the option to get surface data from one single c… ▽ More Flying Triangulation sensors enable a free-hand and motion-robust 3D data acquisition of complex shaped objects. The measurement principle is based on a multi-line light-sectioning approach and uses sophisticated algorithms for real-time registration (S. Ettl et al., Appl. Opt. 51 (2012) 281-289). As "single-shot principle", light sectioning enables the option to get surface data from one single camera exposure. But there is a drawback: A pixel-dense measurement is not possible because of fundamental information-theoretical reasons. By "pixel-dense" we understand that each pixel displays individually measured distance information, neither interpolated from its neighbour pixels nor using lateral context information. Hence, for monomodal single-shot principles, the 3D data generated from one 2D raw image display a significantly lower space-bandwidth than the camera permits. This is the price one must pay for motion robustness. Currently, our sensors project about 10 lines (each with 1000 pixels), reaching an considerable lower data efficiency than theoretically possible for a single-shot sensor. Our aim is to push Flying Triangulation to its information-theoretical limits. Therefore, the line density as well as the measurement depth needs to be significantly increased. This causes serious indexing ambiguities. On the road to a single-shot 3D movie camera, we are working on solutions to overcome the problem of false line indexing by utilizing yet unexploited information. We will present several approaches and will discuss profound information-theoretical questions about the information efficiency of 3D sensors. △ Less

Submitted 17 May, 2013; originally announced May 2013.

Comments: Proceedings of the 7th International Fringe Workshop on Advanced Optical Imaging and Metrology, 2013, Nürtingen, Germany