-
Video Denoising in Fluorescence Guided Surgery
Authors:
Trevor Seets,
Andreas Velten
Abstract:
Fluorescence guided surgery (FGS) is a promising surgical technique that gives surgeons a unique view of tissue that is used to guide their practice by delineating tissue types and diseased areas. As new fluorescent contrast agents are developed that have low fluorescent photon yields, it becomes increasingly important to develop computational models to allow FGS systems to maintain good video qua…
▽ More
Fluorescence guided surgery (FGS) is a promising surgical technique that gives surgeons a unique view of tissue that is used to guide their practice by delineating tissue types and diseased areas. As new fluorescent contrast agents are developed that have low fluorescent photon yields, it becomes increasingly important to develop computational models to allow FGS systems to maintain good video quality in real time environments. To further complicate this task, FGS has a difficult bias noise term from laser leakage light (LLL) that represents unfiltered excitation light that can be on the order of the fluorescent signal. Most conventional video denoising methods focus on zero mean noise, and non-causal processing, both of which are violated in FGS. Luckily in FGS, often a co-located reference video is also captured which we use to simulate the LLL and assist in the denoising processes. In this work, we propose an accurate noise simulation pipeline that includes LLL and propose three baseline deep learning based algorithms for FGS video denoising.
△ Less
Submitted 14 November, 2024;
originally announced November 2024.
-
An approach to Physics Based Single Photon Vision System Design
Authors:
Yizhou Lu,
Trevor Seets,
Felipe Gutierrez-Barragan,
Ehsan Ahmadi,
Andreas Velten
Abstract:
The design of the camera and optical measurement is a crucial part of optimizing machine vision systems. However, camera designs are usually optimized to produce human-interpretable images. Moreover, camera optimization typically makes the assumption of additive noise, while modern optical imaging systems are mainly affected by photon noise which is not additive. Previous studies have highlighted…
▽ More
The design of the camera and optical measurement is a crucial part of optimizing machine vision systems. However, camera designs are usually optimized to produce human-interpretable images. Moreover, camera optimization typically makes the assumption of additive noise, while modern optical imaging systems are mainly affected by photon noise which is not additive. Previous studies have highlighted the fundamental effect of the noise model on the outcome of the design process in the context of coded single-pixel and compressed sensing cameras.
In addition to the exact noise model, the nature of the data encountered and the vision task fundamentally affect the optimal hardware design. This means that a camera or compressed sensing code that is optimal for capturing images is almost certainly not optimal for any downstream vision task. In this work, we used a simple end-to-end model on a single-pixel camera to study the effects of the noise model for different vision tasks. We demonstrate the importance of incorporating non-linear noise features in optimizing even the simplest vision systems.
△ Less
Submitted 19 February, 2025; v1 submitted 27 July, 2023;
originally announced July 2023.
-
Passive Inter-Photon Imaging
Authors:
Atul Ingle,
Trevor Seets,
Mauro Buttafava,
Shantanu Gupta,
Alberto Tosi,
Mohit Gupta,
Andreas Velten
Abstract:
Digital camera pixels measure image intensities by converting incident light energy into an analog electrical current, and then digitizing it into a fixed-width binary representation. This direct measurement method, while conceptually simple, suffers from limited dynamic range and poor performance under extreme illumination -- electronic noise dominates under low illumination, and pixel full-well…
▽ More
Digital camera pixels measure image intensities by converting incident light energy into an analog electrical current, and then digitizing it into a fixed-width binary representation. This direct measurement method, while conceptually simple, suffers from limited dynamic range and poor performance under extreme illumination -- electronic noise dominates under low illumination, and pixel full-well capacity results in saturation under bright illumination. We propose a novel intensity cue based on measuring inter-photon timing, defined as the time delay between detection of successive photons. Based on the statistics of inter-photon times measured by a time-resolved single-photon sensor, we develop theory and algorithms for a scene brightness estimator which works over extreme dynamic range; we experimentally demonstrate imaging scenes with a dynamic range of over ten million to one. The proposed techniques, aided by the emergence of single-photon sensors such as single-photon avalanche diodes (SPADs) with picosecond timing resolution, will have implications for a wide range of imaging applications: robotics, consumer photography, astronomy, microscopy and biomedical imaging.
△ Less
Submitted 10 April, 2021; v1 submitted 31 March, 2021;
originally announced April 2021.
-
Motion Adaptive Deblurring with Single-Photon Cameras
Authors:
Trevor Seets,
Atul Ingle,
Martin Laurenzis,
Andreas Velten
Abstract:
Single-photon avalanche diodes (SPADs) are a rapidly developing image sensing technology with extreme low-light sensitivity and picosecond timing resolution. These unique capabilities have enabled SPADs to be used in applications like LiDAR, non-line-of-sight imaging and fluorescence microscopy that require imaging in photon-starved scenarios. In this work we harness these capabilities for dealing…
▽ More
Single-photon avalanche diodes (SPADs) are a rapidly developing image sensing technology with extreme low-light sensitivity and picosecond timing resolution. These unique capabilities have enabled SPADs to be used in applications like LiDAR, non-line-of-sight imaging and fluorescence microscopy that require imaging in photon-starved scenarios. In this work we harness these capabilities for dealing with motion blur in a passive imaging setting in low illumination conditions. Our key insight is that the data captured by a SPAD array camera can be represented as a 3D spatio-temporal tensor of photon detection events which can be integrated along arbitrary spatio-temporal trajectories with dynamically varying integration windows, depending on scene motion. We propose an algorithm that estimates pixel motion from photon timestamp data and dynamically adapts the integration windows to minimize motion blur. Our simulation results show the applicability of this algorithm to a variety of motion profiles including translation, rotation and local object motion. We also demonstrate the real-world feasibility of our method on data captured using a 32x32 SPAD camera.
△ Less
Submitted 14 December, 2020;
originally announced December 2020.
