-
Energy-Efficient Autonomous Aerial Navigation with Dynamic Vision Sensors: A Physics-Guided Neuromorphic Approach
Authors:
Sourav Sanyal,
Amogh Joshi,
Manish Nagaraj,
Rohan Kumar Manna,
Kaushik Roy
Abstract:
Vision-based object tracking is a critical component for achieving autonomous aerial navigation, particularly for obstacle avoidance. Neuromorphic Dynamic Vision Sensors (DVS) or event cameras, inspired by biological vision, offer a promising alternative to conventional frame-based cameras. These cameras can detect changes in intensity asynchronously, even in challenging lighting conditions, with…
▽ More
Vision-based object tracking is a critical component for achieving autonomous aerial navigation, particularly for obstacle avoidance. Neuromorphic Dynamic Vision Sensors (DVS) or event cameras, inspired by biological vision, offer a promising alternative to conventional frame-based cameras. These cameras can detect changes in intensity asynchronously, even in challenging lighting conditions, with a high dynamic range and resistance to motion blur. Spiking neural networks (SNNs) are increasingly used to process these event-based signals efficiently and asynchronously. Meanwhile, physics-based artificial intelligence (AI) provides a means to incorporate system-level knowledge into neural networks via physical modeling. This enhances robustness, energy efficiency, and provides symbolic explainability. In this work, we present a neuromorphic navigation framework for autonomous drone navigation. The focus is on detecting and navigating through moving gates while avoiding collisions. We use event cameras for detecting moving objects through a shallow SNN architecture in an unsupervised manner. This is combined with a lightweight energy-aware physics-guided neural network (PgNN) trained with depth inputs to predict optimal flight times, generating near-minimum energy paths. The system is implemented in the Gazebo simulator and integrates a sensor-fused vision-to-planning neuro-symbolic framework built with the Robot Operating System (ROS) middleware. This work highlights the future potential of integrating event-based vision with physics-guided planning for energy-efficient autonomous navigation, particularly for low-latency decision-making.
△ Less
Submitted 23 April, 2025; v1 submitted 9 February, 2025;
originally announced February 2025.
-
TOFFE -- Temporally-binned Object Flow from Events for High-speed and Energy-Efficient Object Detection and Tracking
Authors:
Adarsh Kumar Kosta,
Amogh Joshi,
Arjun Roy,
Rohan Kumar Manna,
Manish Nagaraj,
Kaushik Roy
Abstract:
Object detection and tracking is an essential perception task for enabling fully autonomous navigation in robotic systems. Edge robot systems such as small drones need to execute complex maneuvers at high-speeds with limited resources, which places strict constraints on the underlying algorithms and hardware. Traditionally, frame-based cameras are used for vision-based perception due to their rich…
▽ More
Object detection and tracking is an essential perception task for enabling fully autonomous navigation in robotic systems. Edge robot systems such as small drones need to execute complex maneuvers at high-speeds with limited resources, which places strict constraints on the underlying algorithms and hardware. Traditionally, frame-based cameras are used for vision-based perception due to their rich spatial information and simplified synchronous sensing capabilities. However, obtaining detailed information across frames incurs high energy consumption and may not even be required. In addition, their low temporal resolution renders them ineffective in high-speed motion scenarios. Event-based cameras offer a biologically-inspired solution to this by capturing only changes in intensity levels at exceptionally high temporal resolution and low power consumption, making them ideal for high-speed motion scenarios. However, their asynchronous and sparse outputs are not natively suitable with conventional deep learning methods. In this work, we propose TOFFE, a lightweight hybrid framework for performing event-based object motion estimation (including pose, direction, and speed estimation), referred to as Object Flow. TOFFE integrates bio-inspired Spiking Neural Networks (SNNs) and conventional Analog Neural Networks (ANNs), to efficiently process events at high temporal resolutions while being simple to train. Additionally, we present a novel event-based synthetic dataset involving high-speed object motion to train TOFFE. Our experimental results show that TOFFE achieves 5.7x/8.3x reduction in energy consumption and 4.6x/5.8x reduction in latency on edge GPU(Jetson TX2)/hybrid hardware(Loihi-2 and Jetson TX2), compared to previous event-based object detection baselines.
△ Less
Submitted 21 January, 2025;
originally announced January 2025.
-
EV-Planner: Energy-Efficient Robot Navigation via Event-Based Physics-Guided Neuromorphic Planner
Authors:
Sourav Sanyal,
Rohan Kumar Manna,
Kaushik Roy
Abstract:
Vision-based object tracking is an essential precursor to performing autonomous aerial navigation in order to avoid obstacles. Biologically inspired neuromorphic event cameras are emerging as a powerful alternative to frame-based cameras, due to their ability to asynchronously detect varying intensities (even in poor lighting conditions), high dynamic range, and robustness to motion blur. Spiking…
▽ More
Vision-based object tracking is an essential precursor to performing autonomous aerial navigation in order to avoid obstacles. Biologically inspired neuromorphic event cameras are emerging as a powerful alternative to frame-based cameras, due to their ability to asynchronously detect varying intensities (even in poor lighting conditions), high dynamic range, and robustness to motion blur. Spiking neural networks (SNNs) have gained traction for processing events asynchronously in an energy-efficient manner. On the other hand, physics-based artificial intelligence (AI) has gained prominence recently, as they enable embedding system knowledge via physical modeling inside traditional analog neural networks (ANNs). In this letter, we present an event-based physics-guided neuromorphic planner (EV-Planner) to perform obstacle avoidance using neuromorphic event cameras and physics-based AI. We consider the task of autonomous drone navigation where the mission is to detect moving gates and fly through them while avoiding a collision. We use event cameras to perform object detection using a shallow spiking neural network in an unsupervised fashion. Utilizing the physical equations of the brushless DC motors present in the drone rotors, we train a lightweight energy-aware physics-guided neural network (PgNN) with depth inputs. This predicts the optimal flight time responsible for generating near-minimum energy paths. We spawn the drone in the Gazebo simulator and implement a sensor-fused vision-to-planning neuro-symbolic framework using Robot Operating System (ROS). Simulation results for safe collision-free flight trajectories are presented with performance analysis, ablation study and potential future research directions
△ Less
Submitted 3 January, 2024; v1 submitted 21 July, 2023;
originally announced July 2023.
-
Demixing in binary mixtures with differential diffusivity at high density
Authors:
Erin McCarthy,
Ojan Damavandi,
Raj Kumar Manna,
M. Lisa Manning
Abstract:
Spontaneous phase separation, or demixing, is important in biological phenomena such as cell sorting. In particle-based models, an open question is whether differences in diffusivity can drive such demixing. While differential-diffusivity-induced phase separation occurs in mixtures with a packing fraction up to $0.7$ [Weber et al. Phys Rev Lett 2016], here we investigate whether demixing persists…
▽ More
Spontaneous phase separation, or demixing, is important in biological phenomena such as cell sorting. In particle-based models, an open question is whether differences in diffusivity can drive such demixing. While differential-diffusivity-induced phase separation occurs in mixtures with a packing fraction up to $0.7$ [Weber et al. Phys Rev Lett 2016], here we investigate whether demixing persists at even higher densities relevant for cells. For particle packing fractions between $0.7$ and $1.0$ the system demixes, but at packing fractions above unity the system remains mixed, exposing re-entrant behavior in the phase diagram. We also find that a confluent Voronoi model for tissues does not phase separate, consistent with the highest-density particle-based simulations.
△ Less
Submitted 6 July, 2023;
originally announced July 2023.
-
Towards End-to-End Deep Learning for Autonomous Racing: On Data Collection and a Unified Architecture for Steering and Throttle Prediction
Authors:
Shakti N. Wadekar,
Benjamin J. Schwartz,
Shyam S. Kannan,
Manuel Mar,
Rohan Kumar Manna,
Vishnu Chellapandi,
Daniel J. Gonzalez,
Aly El Gamal
Abstract:
Deep Neural Networks (DNNs) which are trained end-to-end have been successfully applied to solve complex problems that we have not been able to solve in past decades. Autonomous driving is one of the most complex problems which is yet to be completely solved and autonomous racing adds more complexity and exciting challenges to this problem. Towards the challenge of applying end-to-end learning to…
▽ More
Deep Neural Networks (DNNs) which are trained end-to-end have been successfully applied to solve complex problems that we have not been able to solve in past decades. Autonomous driving is one of the most complex problems which is yet to be completely solved and autonomous racing adds more complexity and exciting challenges to this problem. Towards the challenge of applying end-to-end learning to autonomous racing, this paper shows results on two aspects: (1) Analyzing the relationship between the driving data used for training and the maximum speed at which the DNN can be successfully applied for predicting steering angle, (2) Neural network architecture and training methodology for learning steering and throttle without any feedback or recurrent connections.
△ Less
Submitted 4 May, 2021;
originally announced May 2021.
-
Colloidal transport by active filaments
Authors:
Raj Kumar Manna,
P. B. Sunil Kumar,
R. Adhikari
Abstract:
Enhanced colloidal transport beyond the limit imposed by diffusion is usually achieved through external fields. Here, we demonstrate the ballistic transport of a colloidal sphere using internal sources of energy provided by an attached active filament. The latter is modeled as a chain of chemo-mechanically active beads connected by potentials that enforce semi-flexibility and self-avoidance. The f…
▽ More
Enhanced colloidal transport beyond the limit imposed by diffusion is usually achieved through external fields. Here, we demonstrate the ballistic transport of a colloidal sphere using internal sources of energy provided by an attached active filament. The latter is modeled as a chain of chemo-mechanically active beads connected by potentials that enforce semi-flexibility and self-avoidance. The fluid flow produced by the active beads and the forces they mediate are explicitly taken into account in the overdamped equations of motion describing the colloid-filament assembly. The speed and efficiency of transport depend on the dynamical conformational states of the filament. We characterize these states using filament writhe as an order parameter and identify ones yielding maxima in speed and efficiency of transport. The transport mechanism reported here has a remarkable resemblance to the flagellar propulsion of microorganisms which suggests its utility in biomimetic systems.
△ Less
Submitted 24 December, 2016;
originally announced December 2016.
-
Excess energy of an ultracold Fermi gas in a trapped geometry
Authors:
Shyamal Biswas,
Debnarayan Jana,
Raj Kumar Manna
Abstract:
We have analytically explored finite size and interparticle interaction corrections to the average energy of a harmonically trapped Fermi gas below and above the Fermi temperature, and have obtained a better fitting for the excess energy reported by DeMarco and Jin [Science $\textbf{285}$, 1703 (1999)]. We have presented a perturbative calculation within a mean field approximation.
We have analytically explored finite size and interparticle interaction corrections to the average energy of a harmonically trapped Fermi gas below and above the Fermi temperature, and have obtained a better fitting for the excess energy reported by DeMarco and Jin [Science $\textbf{285}$, 1703 (1999)]. We have presented a perturbative calculation within a mean field approximation.
△ Less
Submitted 7 June, 2012; v1 submitted 25 April, 2011;
originally announced April 2011.
