-
Invariant neuromorphic representations of tactile stimuli improve robustness of a real-time texture classification system
Authors:
Mark M. Iskarous,
Zan Chaudhry,
Fangjie Li,
Samuel Bello,
Sriramana Sankar,
Ariel Slepyan,
Natasha Chugh,
Christopher L. Hunt,
Rebecca J. Greene,
Nitish V. Thakor
Abstract:
Humans have an exquisite sense of touch which robotic and prosthetic systems aim to recreate. We developed algorithms to create neuron-like (neuromorphic) spiking representations of texture that are invariant to the scanning speed and contact force applied in the sensing process. The spiking representations are based on mimicking activity from mechanoreceptors in human skin and further processing…
▽ More
Humans have an exquisite sense of touch which robotic and prosthetic systems aim to recreate. We developed algorithms to create neuron-like (neuromorphic) spiking representations of texture that are invariant to the scanning speed and contact force applied in the sensing process. The spiking representations are based on mimicking activity from mechanoreceptors in human skin and further processing up to the brain. The neuromorphic encoding process transforms analog sensor readings into speed and force invariant spiking representations in three sequential stages: the force invariance module (in the analog domain), the spiking activity encoding module (transforms from analog to spiking domain), and the speed invariance module (in the spiking domain). The algorithms were tested on a tactile texture dataset collected in 15 speed-force conditions. An offline texture classification system built on the invariant representations has higher classification accuracy, improved computational efficiency, and increased capability to identify textures explored in novel speed-force conditions. The speed invariance algorithm was adapted to a real-time human-operated texture classification system. Similarly, the invariant representations improved classification accuracy, computational efficiency, and capability to identify textures explored in novel conditions. The invariant representation is even more crucial in this context due to human imprecision which seems to the classification system as a novel condition. These results demonstrate that invariant neuromorphic representations enable better performing neurorobotic tactile sensing systems. Furthermore, because the neuromorphic representations are based on biological processing, this work can be used in the future as the basis for naturalistic sensory feedback for upper limb amputees.
△ Less
Submitted 25 November, 2024;
originally announced November 2024.
-
SensoPatch: A Reconfigurable Haptic Feedback with High-Density Tactile Sensing Glove
Authors:
Yanisa Angkanapiwat,
Ariel Slepyan,
Rebecca J. Greene,
Nitish Thakor
Abstract:
Haptic feedback is integral to the improved experience of prosthetic users and the reduction in prosthesis rejection. Prior studies have explored various methods to encode tactile information and deliver vibration feedback. However, a comprehensive study comparing performance across different stimulation locations and feedback modalities for wearable devices is absent and there is no test platform…
▽ More
Haptic feedback is integral to the improved experience of prosthetic users and the reduction in prosthesis rejection. Prior studies have explored various methods to encode tactile information and deliver vibration feedback. However, a comprehensive study comparing performance across different stimulation locations and feedback modalities for wearable devices is absent and there is no test platform. This paper proposes an open-source reconfigurable haptic feedback system which incorporates 25 sensors and wireless communication to allow customized number of vibration motors, adjustable motor placement, and programmable encoding of tactile data to change feedback modalities. To demonstrate potential studies that can be investigated using SensoPatch, we conducted two experiments: 1) to assess the vibration discrimination accuracy on 3 body parts 2) to assess the effect of 6 methods of mapping tactile data to varying number of motors on object manipulation. SensoPatch utilizes low-cost off-the-shelf components, enabling large-scale comparative studies of feedback modalities and stimulation sites to optimize vibrotactile feedback and facilitate its deployment in upper limb prostheses.
△ Less
Submitted 27 September, 2024;
originally announced September 2024.
-
EventLFM: Event Camera integrated Fourier Light Field Microscopy for Ultrafast 3D imaging
Authors:
Ruipeng Guo,
Qianwan Yang,
Andrew S. Chang,
Guorong Hu,
Joseph Greene,
Christopher V. Gabel,
Sixian You,
Lei Tian
Abstract:
Ultrafast 3D imaging is indispensable for visualizing complex and dynamic biological processes. Conventional scanning-based techniques necessitate an inherent trade-off between acquisition speed and space-bandwidth product (SBP). Emerging single-shot 3D wide-field techniques offer a promising alternative but are bottlenecked by the synchronous readout constraints of conventional CMOS systems, thus…
▽ More
Ultrafast 3D imaging is indispensable for visualizing complex and dynamic biological processes. Conventional scanning-based techniques necessitate an inherent trade-off between acquisition speed and space-bandwidth product (SBP). Emerging single-shot 3D wide-field techniques offer a promising alternative but are bottlenecked by the synchronous readout constraints of conventional CMOS systems, thus restricting data throughput to maintain high SBP at limited frame rates. To address this, we introduce EventLFM, a straightforward and cost-effective system that overcomes these challenges by integrating an event camera with Fourier light field microscopy (LFM), a state-of-the-art single-shot 3D wide-field imaging technique. The event camera operates on a novel asynchronous readout architecture, thereby bypassing the frame rate limitations inherent to conventional CMOS systems. We further develop a simple and robust event-driven LFM reconstruction algorithm that can reliably reconstruct 3D dynamics from the unique spatiotemporal measurements captured by EventLFM. Experimental results demonstrate that EventLFM can robustly reconstruct fast-moving and rapidly blinking 3D fluorescent samples at kHz frame rates. Furthermore, we highlight EventLFM's capability for imaging of blinking neuronal signals in scattering mouse brain tissues and 3D tracking of GFP-labeled neurons in freely moving C. elegans. We believe that the combined ultrafast speed and large 3D SBP offered by EventLFM may open up new possibilities across many biomedical applications.
△ Less
Submitted 3 April, 2024; v1 submitted 1 October, 2023;
originally announced October 2023.
-
Robust single-shot 3D fluorescence imaging in scattering media with a simulator-trained neural network
Authors:
Jeffrey Alido,
Joseph Greene,
Yujia Xue,
Guorong Hu,
Yunzhe Li,
Mitchell Gilmore,
Kevin J. Monk,
Brett T. DiBenedictis,
Ian G. Davison,
Lei Tian
Abstract:
Imaging through scattering is a pervasive and difficult problem in many biological applications. The high background and the exponentially attenuated target signals due to scattering fundamentally limits the imaging depth of fluorescence microscopy. Light-field systems are favorable for high-speed volumetric imaging, but the 2D-to-3D reconstruction is fundamentally ill-posed, and scattering exacer…
▽ More
Imaging through scattering is a pervasive and difficult problem in many biological applications. The high background and the exponentially attenuated target signals due to scattering fundamentally limits the imaging depth of fluorescence microscopy. Light-field systems are favorable for high-speed volumetric imaging, but the 2D-to-3D reconstruction is fundamentally ill-posed, and scattering exacerbates the condition of the inverse problem. Here, we develop a scattering simulator that models low-contrast target signals buried in heterogeneous strong background. We then train a deep neural network solely on synthetic data to descatter and reconstruct a 3D volume from a single-shot light-field measurement with low signal-to-background ratio (SBR). We apply this network to our previously developed Computational Miniature Mesoscope and demonstrate the robustness of our deep learning algorithm on scattering phantoms with different scattering conditions. The network can robustly reconstruct emitters in 3D with a 2D measurement of SBR as low as 1.05 and as deep as a scattering length. We analyze fundamental tradeoffs based on network design factors and out-of-distribution data that affect the deep learning model's generalizability to real experimental data. Broadly, we believe that our simulator-based deep learning approach can be applied to a wide range of imaging through scattering techniques where experimental paired training data is lacking.
△ Less
Submitted 8 December, 2023; v1 submitted 22 March, 2023;
originally announced March 2023.
