Showing 1–13 of 13 results for author: Faccio, D

AI-Enabled sensor fusion of time of flight imaging and mmwave for concealed metal detection

Authors: Chaitanya Kaul, Kevin J. Mitchell, Khaled Kassem, Athanasios Tragakis, Valentin Kapitany, Ilya Starshynov, Federica Villa, Roderick Murray-Smith, Daniele Faccio

Abstract: In the field of detection and ranging, multiple complementary sensing modalities may be used to enrich the information obtained from a dynamic scene. One application of this sensor fusion is in public security and surveillance, whose efficacy and privacy protection measures must be continually evaluated. We present a novel deployment of sensor fusion for the discrete detection of concealed metal o… ▽ More In the field of detection and ranging, multiple complementary sensing modalities may be used to enrich the information obtained from a dynamic scene. One application of this sensor fusion is in public security and surveillance, whose efficacy and privacy protection measures must be continually evaluated. We present a novel deployment of sensor fusion for the discrete detection of concealed metal objects on persons whilst preserving their privacy. This is achieved by coupling off-the-shelf mmWave radar and depth camera technology with a novel neural network architecture that processes the radar signals using convolutional Long Short-term Memory (LSTM) blocks and the depth signal, using convolutional operations. The combined latent features are then magnified using a deep feature magnification to learn cross-modality dependencies in the data. We further propose a decoder, based on the feature extraction and embedding block, to learn an efficient upsampling of the latent space to learn the location of the concealed object in the spatial domain through radar feature guidance. We demonstrate the detection of presence and inference of 3D location of concealed metal objects with an accuracy of up to 95%, using a technique that is robust to multiple persons. This work provides a demonstration of the potential for cost effective and portable sensor fusion, with strong opportunities for further development. △ Less

Submitted 1 August, 2024; originally announced August 2024.

Single-sample image-fusion upsampling of fluorescence lifetime images

Authors: Valentin Kapitány, Areeba Fatima, Vytautas Zickus, Jamie Whitelaw, Ewan McGhee, Robert Insall, Laura Machesky, Daniele Faccio

Abstract: Fluorescence lifetime imaging microscopy (FLIM) provides detailed information about molecular interactions and biological processes. A major bottleneck for FLIM is image resolution at high acquisition speeds, due to the engineering and signal-processing limitations of time-resolved imaging technology. Here we present single-sample image-fusion upsampling (SiSIFUS), a data-fusion approach to comput… ▽ More Fluorescence lifetime imaging microscopy (FLIM) provides detailed information about molecular interactions and biological processes. A major bottleneck for FLIM is image resolution at high acquisition speeds, due to the engineering and signal-processing limitations of time-resolved imaging technology. Here we present single-sample image-fusion upsampling (SiSIFUS), a data-fusion approach to computational FLIM super-resolution that combines measurements from a low-resolution time-resolved detector (that measures photon arrival time) and a high-resolution camera (that measures intensity only). To solve this otherwise ill-posed inverse retrieval problem, we introduce statistically informed priors that encode local and global dependencies between the two single-sample measurements. This bypasses the risk of out-of-distribution hallucination as in traditional data-driven approaches and delivers enhanced images compared for example to standard bilinear interpolation. The general approach laid out by SiSIFUS can be applied to other image super-resolution problems where two different datasets are available. △ Less

Submitted 19 April, 2024; originally announced April 2024.

Comments: 18 pages, 11 figures. To be published in Science Advances

ACM Class: I.4.3; I.4.8; I.5.1

A large-scale multimodal dataset of human speech recognition

Authors: Yao Ge, Chong Tang, Haobo Li, Zikang Zhang, Wenda Li, Kevin Chetty, Daniele Faccio, Qammer H. Abbasi, Muhammad Imran

Abstract: Nowadays, non-privacy small-scale motion detection has attracted an increasing amount of research in remote sensing in speech recognition. These new modalities are employed to enhance and restore speech information from speakers of multiple types of data. In this paper, we propose a dataset contains 7.5 GHz Channel Impulse Response (CIR) data from ultra-wideband (UWB) radars, 77-GHz frequency modu… ▽ More Nowadays, non-privacy small-scale motion detection has attracted an increasing amount of research in remote sensing in speech recognition. These new modalities are employed to enhance and restore speech information from speakers of multiple types of data. In this paper, we propose a dataset contains 7.5 GHz Channel Impulse Response (CIR) data from ultra-wideband (UWB) radars, 77-GHz frequency modulated continuous wave (FMCW) data from millimetre wave (mmWave) radar, and laser data. Meanwhile, a depth camera is adopted to record the landmarks of the subject's lip and voice. Approximately 400 minutes of annotated speech profiles are provided, which are collected from 20 participants speaking 5 vowels, 15 words and 16 sentences. The dataset has been validated and has potential for the research of lip reading and multimodal speech recognition. △ Less

Submitted 14 March, 2023; originally announced March 2023.

mmSense: Detecting Concealed Weapons with a Miniature Radar Sensor

Authors: Kevin Mitchell, Khaled Kassem, Chaitanya Kaul, Valentin Kapitany, Philip Binner, Andrew Ramsay, Roderick Murray-Smith, Daniele Faccio

Abstract: For widespread adoption, public security and surveillance systems must be accurate, portable, compact, and real-time, without impeding the privacy of the individuals being observed. Current systems broadly fall into two categories -- image-based which are accurate, but lack privacy, and RF signal-based, which preserve privacy but lack portability, compactness and accuracy. Our paper proposes mmSen… ▽ More For widespread adoption, public security and surveillance systems must be accurate, portable, compact, and real-time, without impeding the privacy of the individuals being observed. Current systems broadly fall into two categories -- image-based which are accurate, but lack privacy, and RF signal-based, which preserve privacy but lack portability, compactness and accuracy. Our paper proposes mmSense, an end-to-end portable miniaturised real-time system that can accurately detect the presence of concealed metallic objects on persons in a discrete, privacy-preserving modality. mmSense features millimeter wave radar technology, provided by Google's Soli sensor for its data acquisition, and TransDope, our real-time neural network, capable of processing a single radar data frame in 19 ms. mmSense achieves high recognition rates on a diverse set of challenging scenes while running on standard laptop hardware, demonstrating a significant advancement towards creating portable, cost-effective real-time radar based surveillance systems. △ Less

Submitted 28 February, 2023; originally announced February 2023.

Comments: Accepted by ICASSP 2023

Bessel Equivariant Networks for Inversion of Transmission Effects in Multi-Mode Optical Fibres

Authors: Joshua Mitton, Simon Peter Mekhail, Miles Padgett, Daniele Faccio, Marco Aversa, Roderick Murray-Smith

Abstract: We develop a new type of model for solving the task of inverting the transmission effects of multi-mode optical fibres through the construction of an $\mathrm{SO}^{+}(2,1)$-equivariant neural network. This model takes advantage of the of the azimuthal correlations known to exist in fibre speckle patterns and naturally accounts for the difference in spatial arrangement between input and speckle pat… ▽ More We develop a new type of model for solving the task of inverting the transmission effects of multi-mode optical fibres through the construction of an $\mathrm{SO}^{+}(2,1)$-equivariant neural network. This model takes advantage of the of the azimuthal correlations known to exist in fibre speckle patterns and naturally accounts for the difference in spatial arrangement between input and speckle patterns. In addition, we use a second post-processing network to remove circular artifacts, fill gaps, and sharpen the images, which is required due to the nature of optical fibre transmission. This two stage approach allows for the inspection of the predicted images produced by the more robust physically motivated equivariant model, which could be useful in a safety-critical application, or by the output of both models, which produces high quality images. Further, this model can scale to previously unachievable resolutions of imaging with multi-mode optical fibres and is demonstrated on $256 \times 256$ pixel images. This is a result of improving the trainable parameter requirement from $\mathcal{O}(N^4)$ to $\mathcal{O}(m)$, where $N$ is pixel size and $m$ is number of fibre modes. Finally, this model generalises to new images, outside of the set of training data classes, better than previous models. △ Less

Submitted 17 October, 2022; v1 submitted 26 July, 2022; originally announced July 2022.

Comments: NeurIPS 2022

Super-Resolution Time-Resolved Imaging using Computational Sensor Fusion

Authors: C. Callenberg, A. Lyons, D. den Brok, A. Fatima, A. Turpin, V. Zickus, L. Machesky, J. Whitelaw, D. Faccio, M. B. Hullin

Abstract: Imaging across both the full transverse spatial and temporal dimensions of a scene with high precision in all three coordinates is key to applications ranging from LIDAR to fluorescence lifetime imaging. However, compromises that sacrifice, for example, spatial resolution at the expense of temporal resolution are often required, in particular when the full 3-dimensional data cube is required in sh… ▽ More Imaging across both the full transverse spatial and temporal dimensions of a scene with high precision in all three coordinates is key to applications ranging from LIDAR to fluorescence lifetime imaging. However, compromises that sacrifice, for example, spatial resolution at the expense of temporal resolution are often required, in particular when the full 3-dimensional data cube is required in short acquisition times. We introduce a sensor fusion approach that combines data having low-spatial resolution but high temporal precision gathered with a single-photon-avalanche-diode (SPAD) array with set of data that has high spatial but no temporal resolution, such as that acquired with a standard CMOS camera. Our method, based on blurring the image on the SPAD array and computational sensor fusion, reconstructs time-resolved images at significantly higher spatial resolution than the SPAD input, upsampling numerical data by a factor 12x12, and demonstrating up to 4x4 upsampling of experimental data. We demonstrate the technique for both LIDAR applications and FLIM of fluorescent cancer cells. This technique paves the way to high spatial resolution SPAD imaging or, equivalently, FLIM imaging with conventional microscopes at frame rates accelerated by more than an order of magnitude. △ Less

Submitted 8 January, 2021; originally announced January 2021.

Comments: 8 pages, 4 figures

3D imaging from multipath temporal echoes

Authors: Alex Turpin, Valentin Kapitany, Jack Radford, Davide Rovelli, Kevin Mitchell, Ashley Lyons, Ilya Starshynov, Daniele Faccio

Abstract: Echo-location is a broad approach to imaging and sensing that includes both man-made RADAR, LIDAR, SONAR and also animal navigation. However, full 3D information based on echo-location requires some form of scanning of the scene in order to provide the spatial location of the echo origin-points. Without this spatial information, imaging objects in 3D is a very challenging task as the inverse retri… ▽ More Echo-location is a broad approach to imaging and sensing that includes both man-made RADAR, LIDAR, SONAR and also animal navigation. However, full 3D information based on echo-location requires some form of scanning of the scene in order to provide the spatial location of the echo origin-points. Without this spatial information, imaging objects in 3D is a very challenging task as the inverse retrieval problem is strongly ill-posed. Here, we show that the temporal information encoded in the return echoes that are reflected multiple times within a scene is sufficient to faithfully render an image in 3D. Numerical modelling and an information theoretic perspective prove the concept and provide insight into the role of the multipath information. We experimentally demonstrate the concept by using both radio-frequency and acoustic waves for imaging individuals moving in a closed environment. △ Less

Submitted 15 June, 2021; v1 submitted 17 November, 2020; originally announced November 2020.

Comments: Main document: 5 pages, 3 figures. Supplementary document: 8 pages, 7 figures. Supplementary videos can be accessed in the following link: https://www.youtube.com/playlist?list=PLqMUzW5Nvp3RhHK1O4k34NVIbfeAKiVbC

Journal ref: Phys. Rev. Lett. 126, 174301 (2021)

Statistical Dependencies Beyond Linear Correlations in Light Scattered by Disordered Media

Authors: Ilya Starshynov, Alex Turpin, Philip Binner, Daniele Faccio

Abstract: Imaging through scattering and random media is an outstanding problem that to date has been tackled by either measuring the medium transmission matrix or exploiting linear correlations in the transmitted speckle patterns. However, transmission matrix techniques require interferometric stability and linear correlations, such as the memory effect, can be exploited only in thin scattering media. Here… ▽ More Imaging through scattering and random media is an outstanding problem that to date has been tackled by either measuring the medium transmission matrix or exploiting linear correlations in the transmitted speckle patterns. However, transmission matrix techniques require interferometric stability and linear correlations, such as the memory effect, can be exploited only in thin scattering media. Here we show the existence of a statistical dependency in strongly scattered optical fields in a case where first-order correlations are not expected. We also show that this statistical dependence and the related information transport is directly linked to artificial neural network imaging in strongly scattering, dynamic media. These non-trivial dependencies provide a key to imaging through dynamic and thick scattering media with applications for deep-tissue imaging or imaging through smoke or fog △ Less

Submitted 6 April, 2022; v1 submitted 16 November, 2020; originally announced November 2020.

The role of late photons in diffuse optical imaging

Authors: Jack Radford, Ashley Lyons, Francesco Tonolini, Daniele Faccio

Abstract: The ability to image through turbid media such as organic tissues, is a highly attractive prospect for biological and medical imaging. This is challenging however, due to the highly scattering properties of tissues which scramble the image information. The earliest photons that arrive at the detector are often associated with ballistic transmission, whilst the later photons are associated with com… ▽ More The ability to image through turbid media such as organic tissues, is a highly attractive prospect for biological and medical imaging. This is challenging however, due to the highly scattering properties of tissues which scramble the image information. The earliest photons that arrive at the detector are often associated with ballistic transmission, whilst the later photons are associated with complex paths due to multiple independent scattering events and are therefore typically considered to be detrimental to the final image formation process. In this work we report on the importance of these highly diffuse, "late" photons for computational time-of-flight diffuse optical imaging. In thick scattering materials, >80 transport mean free paths, we provide evidence that including late photons in the inverse retrieval enhances the image reconstruction quality. We also show that the late photons alone have sufficient information to retrieve images of a similar quality to early photon gated data. This result emphasises the importance in the strongly diffusive regime discussed here, of fully time-resolved imaging techniques. △ Less

Submitted 24 August, 2020; originally announced August 2020.

Comments: 17 pages, 5 figures

Journal ref: Opt. Express 28, 29486-29495 (2020)

Non-line-of-sight Imaging

Authors: D. Faccio, A. Velten, G. Wetzstein

Abstract: Emerging single-photon-sensitive sensors combined with advanced inverse methods to process picosecond-accurate time-stamped photon counts have given rise to unprecedented imaging capabilities. Rather than imaging photons that travel along direct paths from a source to an object and back to the detector, non-line-of-sight (NLOS) imaging approaches analyse photons {scattered from multiple surfaces t… ▽ More Emerging single-photon-sensitive sensors combined with advanced inverse methods to process picosecond-accurate time-stamped photon counts have given rise to unprecedented imaging capabilities. Rather than imaging photons that travel along direct paths from a source to an object and back to the detector, non-line-of-sight (NLOS) imaging approaches analyse photons {scattered from multiple surfaces that travel} along indirect light paths to estimate 3D images of scenes outside the direct line of sight of a camera, hidden by a wall or other obstacles. Here we review recent advances in the field of NLOS imaging, discussing how to see around corners and future prospects for the field. △ Less

Submitted 16 May, 2020; originally announced May 2020.

Journal ref: Nature Review Physics (2020)

Spatial images from temporal data

Authors: Alex Turpin, Gabriella Musarra, Valentin Kapitany, Francesco Tonolini, Ashley Lyons, Ilya Starshynov, Federica Villa, Enrico Conca, Francesco Fioranelli, Roderick Murray-Smith, Daniele Faccio

Abstract: Traditional paradigms for imaging rely on the use of a spatial structure, either in the detector (pixels arrays) or in the illumination (patterned light). Removal of the spatial structure in the detector or illumination, i.e., imaging with just a single-point sensor, would require solving a very strongly ill-posed inverse retrieval problem that to date has not been solved. Here, we demonstrate a d… ▽ More Traditional paradigms for imaging rely on the use of a spatial structure, either in the detector (pixels arrays) or in the illumination (patterned light). Removal of the spatial structure in the detector or illumination, i.e., imaging with just a single-point sensor, would require solving a very strongly ill-posed inverse retrieval problem that to date has not been solved. Here, we demonstrate a data-driven approach in which full 3D information is obtained with just a single-point, single-photon avalanche diode that records the arrival time of photons reflected from a scene that is illuminated with short pulses of light. Imaging with single-point time-of-flight (temporal) data opens new routes in terms of speed, size, and functionality. As an example, we show how the training based on an optical time-of-flight camera enables a compact radio-frequency impulse radio detection and ranging transceiver to provide 3D images. △ Less

Submitted 4 August, 2020; v1 submitted 2 December, 2019; originally announced December 2019.

Comments: This is the final version as published in Optica Vol. 7, Issue 8, pp. 900-905 (2020)

Transmission of natural scene images through a multimode fibre

Authors: Piergiorgio Caramazza, Oisín Moran, Roderick Murray-Smith, Daniele Faccio

Abstract: The optical transport of images through a multimode fibre remains an outstanding challenge with applications ranging from optical communications to neuro-imaging. State of the art approaches either involve measurement and control of the full complex field transmitted through the fibre or, more recently, training of artificial neural networks that however, are typically limited to image classes bel… ▽ More The optical transport of images through a multimode fibre remains an outstanding challenge with applications ranging from optical communications to neuro-imaging. State of the art approaches either involve measurement and control of the full complex field transmitted through the fibre or, more recently, training of artificial neural networks that however, are typically limited to image classes belong to the same class as the training data set. Here we implement a method that statistically reconstructs the inverse transformation matrix for the fibre. We demonstrate imaging at high frame rates, high resolutions and in full colour of natural scenes, thus demonstrating general-purpose imaging capability. Real-time imaging over long fibre lengths opens alternative routes to exploitation for example for secure communication systems, novel remote imaging devices, quantum state control processing and endoscopy. △ Less

Submitted 26 April, 2019; originally announced April 2019.

Non-line-of-sight 3D imaging with a single-pixel camera

Authors: G. Musarra, A. Lyons, E. Conca, Y. Altmann, F. Villa, F. Zappa, M. J. Padgett, D. Faccio

Abstract: Real time, high resolution 3D reconstruction of scenes hidden from the direct field of view is a challenging field of research with applications in real-life situations related e.g. to surveillance, self-driving cars and rescue missions. Most current techniques recover the 3D structure of a non-lineof-sight (NLOS) static scene by detecting the return signal from the hidden object on a scattering o… ▽ More Real time, high resolution 3D reconstruction of scenes hidden from the direct field of view is a challenging field of research with applications in real-life situations related e.g. to surveillance, self-driving cars and rescue missions. Most current techniques recover the 3D structure of a non-lineof-sight (NLOS) static scene by detecting the return signal from the hidden object on a scattering observation area. Here, we demonstrate the full colour retrieval of the 3D shape of a hidden scene by coupling back-projection imaging algorithms with the high-resolution time-of-flight information provided by a single-pixel camera. By using a high effciency Single-Photon Avalanche Diode (SPAD) detector, this technique provides the advantage of imaging with no mechanical scanning parts, with acquisition times down to sub-seconds. △ Less

Submitted 12 March, 2019; originally announced March 2019.

Comments: 6 pages, 4 figures

Journal ref: Phys. Rev. Applied 12, 011002 (2019)