-
Adaptive Spiking with Plasticity for Energy Aware Neuromorphic Systems
Authors:
Eduardo Calle-Ortiz,
Hui Guan,
Deepak Ganesan,
Phuc Nguyen
Abstract:
This paper presents ASPEN, a novel energy-aware technique for neuromorphic systems that could unleash the future of intelligent, always-on, ultra-low-power, and low-burden wearables. Our main research objectives are to explore the feasibility of neuromorphic computing for wearables, identify open research directions, and demonstrate the feasibility of developing an adaptive spiking technique for e…
▽ More
This paper presents ASPEN, a novel energy-aware technique for neuromorphic systems that could unleash the future of intelligent, always-on, ultra-low-power, and low-burden wearables. Our main research objectives are to explore the feasibility of neuromorphic computing for wearables, identify open research directions, and demonstrate the feasibility of developing an adaptive spiking technique for energy-aware computation, which can be game-changing for resource-constrained devices in always-on applications. As neuromorphic computing systems operate based on spike events, their energy consumption is closely related to spiking activity, i.e., each spike incurs computational and power costs; consequently, minimizing the number of spikes is a critical strategy for operating under constrained energy budgets. To support this goal, ASPEN utilizes stochastic perturbations to the neuronal threshold during training to not only enhance the network's robustness across varying thresholds, which can be controlled at inference time, but also act as a regularizer that improves generalization, reduces spiking activity, and enables energy control without the need for complex retraining or pruning. More specifically, ASPEN adaptively adjusts intrinsic neuronal parameters as a lightweight and scalable technique for dynamic energy control without reconfiguring the entire model. Our evaluation on neuromorphic emulator and hardware shows that ASPEN significantly reduces spike counts and energy consumption while maintaining accuracy comparable to state-of-the-art methods.
△ Less
Submitted 11 August, 2025;
originally announced August 2025.
-
Communication-Efficient Multi-Device Inference Acceleration for Transformer Models
Authors:
Xiao Liu,
Lijun Zhang,
Deepak Ganesan,
Hui Guan
Abstract:
Transformer models power many AI applications but suffer from high inference latency, limiting their use in real-time settings. Multi-device inference can reduce latency by parallelizing computation. Yet, existing methods require high inter-device bandwidth, making them impractical for bandwidth-constrained environments. We propose ASTRA, a communication-efficient framework that accelerates Transf…
▽ More
Transformer models power many AI applications but suffer from high inference latency, limiting their use in real-time settings. Multi-device inference can reduce latency by parallelizing computation. Yet, existing methods require high inter-device bandwidth, making them impractical for bandwidth-constrained environments. We propose ASTRA, a communication-efficient framework that accelerates Transformer inference through a novel integration of sequence parallelism and a Mixed-Precision Attention mechanism designed to minimize inter-device communication. ASTRA compresses non-local token embeddings via vector quantization and preserves task accuracy through two optimizations, Noise-Augmented Quantization and Distributed Class Tokens. Experiments on ViT and GPT2 across vision and NLP tasks show that ASTRA achieves up to 2.64X speedups over single-device inference and up to 15.25X speedups over state-of-the-art multi-device inferences, while operating under bandwidths as low as 10 Mbps. ASTRA is open-sourced at https://github.com/xl1990/Astra.
△ Less
Submitted 25 May, 2025;
originally announced May 2025.
-
Aligned Vector Quantization for Edge-Cloud Collabrative Vision-Language Models
Authors:
Xiao Liu,
Lijun Zhang,
Deepak Ganesan,
Hui Guan
Abstract:
Vision Language Models (VLMs) are central to Visual Question Answering (VQA) systems and are typically deployed in the cloud due to their high computational demands. However, this cloud-only approach underutilizes edge computational resources and requires significant bandwidth for transmitting raw images. In this paper, we introduce an edge-cloud collaborative VQA system, called LLaVA-AlignedVQ, w…
▽ More
Vision Language Models (VLMs) are central to Visual Question Answering (VQA) systems and are typically deployed in the cloud due to their high computational demands. However, this cloud-only approach underutilizes edge computational resources and requires significant bandwidth for transmitting raw images. In this paper, we introduce an edge-cloud collaborative VQA system, called LLaVA-AlignedVQ, which features a novel Aligned Vector Quantization algorithm (AlignedVQ) that efficiently compress intermediate features without compromising accuracy to support partitioned execution. Our experiments demonstrate that LLaVA-AlignedVQ achieves approximately 1365x compression rate of intermediate features, reducing data transmission overhead by 96.8% compared to transmitting JPEG90-compressed images to the cloud. LLaVA-AlignedVQ achieves an inference speedup of 2-15x while maintaining high accuracy, remaining within -2.23% to +1.6% of the original model's accuracy performance across eight VQA datasets, compared to the cloud-only solution.
△ Less
Submitted 8 November, 2024;
originally announced November 2024.
-
In-Situ Fine-Tuning of Wildlife Models in IoT-Enabled Camera Traps for Efficient Adaptation
Authors:
Mohammad Mehdi Rastikerdar,
Jin Huang,
Hui Guan,
Deepak Ganesan
Abstract:
Resource-constrained IoT devices increasingly rely on deep learning models, however, these models experience significant accuracy drops due to domain shifts when encountering variations in lighting, weather, and seasonal conditions. While cloud-based retraining can address this issue, many IoT deployments operate with limited connectivity and energy constraints, making traditional fine-tuning appr…
▽ More
Resource-constrained IoT devices increasingly rely on deep learning models, however, these models experience significant accuracy drops due to domain shifts when encountering variations in lighting, weather, and seasonal conditions. While cloud-based retraining can address this issue, many IoT deployments operate with limited connectivity and energy constraints, making traditional fine-tuning approaches impractical. We explore this challenge through the lens of wildlife ecology, where camera traps must maintain accurate species classification across changing seasons, weather, and habitats without reliable connectivity. We introduce WildFit, an autonomous in-situ adaptation framework that leverages the key insight that background scenes change more frequently than the visual characteristics of monitored species. WildFit combines background-aware synthesis to generate training samples on-device with drift-aware fine-tuning that triggers model updates only when necessary to conserve resources. Our background-aware synthesis surpasses efficient baselines by 7.3\% and diffusion models by 3.0\% while being orders of magnitude faster, our drift-aware fine-tuning achieves Pareto optimality with 50\% fewer updates and 1.5\% higher accuracy, and the end-to-end system outperforms domain adaptation approaches by 20--35%\% while consuming only 11.2 Wh over 37 days -- enabling battery-powered deployment.
△ Less
Submitted 10 August, 2025; v1 submitted 12 September, 2024;
originally announced September 2024.
-
GDTM: An Indoor Geospatial Tracking Dataset with Distributed Multimodal Sensors
Authors:
Ho Lyun Jeong,
Ziqi Wang,
Colin Samplawski,
Jason Wu,
Shiwei Fang,
Lance M. Kaplan,
Deepak Ganesan,
Benjamin Marlin,
Mani Srivastava
Abstract:
Constantly locating moving objects, i.e., geospatial tracking, is essential for autonomous building infrastructure. Accurate and robust geospatial tracking often leverages multimodal sensor fusion algorithms, which require large datasets with time-aligned, synchronized data from various sensor types. However, such datasets are not readily available. Hence, we propose GDTM, a nine-hour dataset for…
▽ More
Constantly locating moving objects, i.e., geospatial tracking, is essential for autonomous building infrastructure. Accurate and robust geospatial tracking often leverages multimodal sensor fusion algorithms, which require large datasets with time-aligned, synchronized data from various sensor types. However, such datasets are not readily available. Hence, we propose GDTM, a nine-hour dataset for multimodal object tracking with distributed multimodal sensors and reconfigurable sensor node placements. Our dataset enables the exploration of several research problems, such as optimizing architectures for processing multimodal data, and investigating models' robustness to adverse sensing conditions and sensor placement variances. A GitHub repository containing the code, sample data, and checkpoints of this work is available at https://github.com/nesl/GDTM.
△ Less
Submitted 21 February, 2024;
originally announced February 2024.
-
Efficient IoT Inference via Context-Awareness
Authors:
Mohammad Mehdi Rastikerdar,
Jin Huang,
Shiwei Fang,
Hui Guan,
Deepak Ganesan
Abstract:
While existing strategies to execute deep learning-based classification on low-power platforms assume the models are trained on all classes of interest, this paper posits that adopting context-awareness i.e. narrowing down a classification task to the current deployment context consisting of only recent inference queries can substantially enhance performance in resource-constrained environments. W…
▽ More
While existing strategies to execute deep learning-based classification on low-power platforms assume the models are trained on all classes of interest, this paper posits that adopting context-awareness i.e. narrowing down a classification task to the current deployment context consisting of only recent inference queries can substantially enhance performance in resource-constrained environments. We propose a new paradigm, CACTUS, for scalable and efficient context-aware classification where a micro-classifier recognizes a small set of classes relevant to the current context and, when context change happens (e.g., a new class comes into the scene), rapidly switches to another suitable micro-classifier. CACTUS features several innovations, including optimizing the training cost of context-aware classifiers, enabling on-the-fly context-aware switching between classifiers, and balancing context switching costs and performance gains via simple yet effective switching policies. We show that CACTUS achieves significant benefits in accuracy, latency, and compute budget across a range of datasets and IoT platforms.
△ Less
Submitted 3 December, 2023; v1 submitted 29 October, 2023;
originally announced October 2023.
-
Heteroskedastic Geospatial Tracking with Distributed Camera Networks
Authors:
Colin Samplawski,
Shiwei Fang,
Ziqi Wang,
Deepak Ganesan,
Mani Srivastava,
Benjamin M. Marlin
Abstract:
Visual object tracking has seen significant progress in recent years. However, the vast majority of this work focuses on tracking objects within the image plane of a single camera and ignores the uncertainty associated with predicted object locations. In this work, we focus on the geospatial object tracking problem using data from a distributed camera network. The goal is to predict an object's tr…
▽ More
Visual object tracking has seen significant progress in recent years. However, the vast majority of this work focuses on tracking objects within the image plane of a single camera and ignores the uncertainty associated with predicted object locations. In this work, we focus on the geospatial object tracking problem using data from a distributed camera network. The goal is to predict an object's track in geospatial coordinates along with uncertainty over the object's location while respecting communication constraints that prohibit centralizing raw image data. We present a novel single-object geospatial tracking data set that includes high-accuracy ground truth object locations and video data from a network of four cameras. We present a modeling framework for addressing this task including a novel backbone model and explore how uncertainty calibration and fine-tuning through a differentiable tracker affect performance.
△ Less
Submitted 4 June, 2023;
originally announced June 2023.
-
Eulerian Phase-based Motion Magnification for High-Fidelity Vital Sign Estimation with Radar in Clinical Settings
Authors:
Md Farhan Tasnim Oshim,
Toral Surti,
Stephanie Carreiro,
Deepak Ganesan,
Suren Jayasuriya,
Tauhidur Rahman
Abstract:
Efficient and accurate detection of subtle motion generated from small objects in noisy environments, as needed for vital sign monitoring, is challenging, but can be substantially improved with magnification. We developed a complex Gabor filter-based decomposition method to amplify phases at different spatial wavelength levels to magnify motion and extract 1D motion signals for fundamental frequen…
▽ More
Efficient and accurate detection of subtle motion generated from small objects in noisy environments, as needed for vital sign monitoring, is challenging, but can be substantially improved with magnification. We developed a complex Gabor filter-based decomposition method to amplify phases at different spatial wavelength levels to magnify motion and extract 1D motion signals for fundamental frequency estimation. The phase-based complex Gabor filter outputs are processed and then used to train machine learning models that predict respiration and heart rate with greater accuracy. We show that our proposed technique performs better than the conventional temporal FFT-based method in clinical settings, such as sleep laboratories and emergency departments, as well for a variety of human postures.
△ Less
Submitted 3 December, 2022;
originally announced December 2022.
-
PhyMask: Robust Sensing of Brain Activity and Physiological Signals During Sleep with an All-textile Eye Mask
Authors:
Soha Rostaminia,
S. Zohreh Homayounfar,
Ali Kiaghadi,
Trisha L. Andrew,
Deepak Ganesan
Abstract:
Clinical-grade wearable sleep monitoring is a challenging problem since it requires concurrently monitoring brain activity, eye movement, muscle activity, cardio-respiratory features and gross body movements. This requires multiple sensors to be worn at different locations as well as uncomfortable adhesives and discrete electronic components to be placed on the head. As a result, existing wearable…
▽ More
Clinical-grade wearable sleep monitoring is a challenging problem since it requires concurrently monitoring brain activity, eye movement, muscle activity, cardio-respiratory features and gross body movements. This requires multiple sensors to be worn at different locations as well as uncomfortable adhesives and discrete electronic components to be placed on the head. As a result, existing wearables either compromise comfort or compromise accuracy in tracking sleep variables. We propose PhyMask, an all-textile sleep monitoring solution that is practical and comfortable for continuous use and that acquires all signals of interest to sleep solely using comfortable textile sensors placed on the head. We show that PhyMask can be used to accurately measure sleep stages and advanced sleep markers such as spindles and k-complexes robustly in the real-world setting. We validate PhyMask against polysomnography and show that it significantly outperforms two commercially-available sleep tracking wearables, Fitbit and Oura Ring.
△ Less
Submitted 11 August, 2021; v1 submitted 12 June, 2021;
originally announced June 2021.
-
Integrating Lexical Knowledge in Word Embeddings using Sprinkling and Retrofitting
Authors:
Aakash Srinivasan,
Harshavardhan Kamarthi,
Devi Ganesan,
Sutanu Chakraborti
Abstract:
Neural network based word embeddings, such as Word2Vec and GloVe, are purely data driven in that they capture the distributional information about words from the training corpus. Past works have attempted to improve these embeddings by incorporating semantic knowledge from lexical resources like WordNet. Some techniques like retrofitting modify word embeddings in the post-processing stage while so…
▽ More
Neural network based word embeddings, such as Word2Vec and GloVe, are purely data driven in that they capture the distributional information about words from the training corpus. Past works have attempted to improve these embeddings by incorporating semantic knowledge from lexical resources like WordNet. Some techniques like retrofitting modify word embeddings in the post-processing stage while some others use a joint learning approach by modifying the objective function of neural networks. In this paper, we discuss two novel approaches for incorporating semantic knowledge into word embeddings. In the first approach, we take advantage of Levy et al's work which showed that using SVD based methods on co-occurrence matrix provide similar performance to neural network based embeddings. We propose a 'sprinkling' technique to add semantic relations to the co-occurrence matrix directly before factorization. In the second approach, WordNet similarity scores are used to improve the retrofitting method. We evaluate the proposed methods in both intrinsic and extrinsic tasks and observe significant improvements over the baselines in many of the datasets.
△ Less
Submitted 23 January, 2020; v1 submitted 14 December, 2019;
originally announced December 2019.
-
Model-based Testing of Mobile Systems -- An Empirical Study on QuizUp Android App
Authors:
Vignir Gudmundsson,
Mikael Lindvall,
Luca Aceto,
Johann Bergthorsson,
Dharmalingam Ganesan
Abstract:
We present an empirical study in which model-based testing (MBT) was applied to a mobile system: the Android client of QuizUp, the largest mobile trivia game in the world. The study shows that traditional MBT approaches based on extended finite-state machines can be used to test a mobile app in an effective and efficient way. Non-trivial defects were detected on a deployed system that has millio…
▽ More
We present an empirical study in which model-based testing (MBT) was applied to a mobile system: the Android client of QuizUp, the largest mobile trivia game in the world. The study shows that traditional MBT approaches based on extended finite-state machines can be used to test a mobile app in an effective and efficient way. Non-trivial defects were detected on a deployed system that has millions of users and was already well tested. The duration of the overall testing effort was of three months, including the construction of the models. Maintaining a single behavioral model for the app was key in order to test it in an efficient way.
△ Less
Submitted 1 June, 2016;
originally announced June 2016.
-
Phosphorene and Transition Metal Dichalcogenide 2D Heterojunctions: Application in Excitonic Solar Cells
Authors:
Vellayappan Dheivanayagam S/O Ganesan,
Chun Zhang,
Yuan Ping Feng,
Lei Shen
Abstract:
Using the first-principles GW-Bethe-Salpeter equation method, here we study the excited-state properties, including quasi-particle band structures and optical spectra, of phosphorene, a two-dimensional (2D) atomic layer of black phosphorus. The quasi-particle band gap of monolayer phosphorene is 2.15 eV and its optical gap is 1.6 eV, which is suitable for excitonic thin film solar cell application…
▽ More
Using the first-principles GW-Bethe-Salpeter equation method, here we study the excited-state properties, including quasi-particle band structures and optical spectra, of phosphorene, a two-dimensional (2D) atomic layer of black phosphorus. The quasi-particle band gap of monolayer phosphorene is 2.15 eV and its optical gap is 1.6 eV, which is suitable for excitonic thin film solar cell applications. Next, this potential application is analysed by considering type-II heterostructures with single layered phosphorene and transition metal dichalcogenides (TMDs). These heterojunctions have a potential maximum power conversion efficiency of up to 12\%, which can be further enhanced to 20\% by strain engineering. Our results show that phosphorene is not only a promising new material for use in nanoscale electronics, but also in optoelectronics.
△ Less
Submitted 27 July, 2015;
originally announced July 2015.
-
Machine Learning of Phonologically Conditioned Noun Declensions For Tamil Morphological Generators
Authors:
K. Rajan,
Dr. V. Ramalingam,
Dr. M. Ganesan
Abstract:
This paper presents machine learning solutions to a practical problem of Natural Language Generation (NLG), particularly the word formation in agglutinative languages like Tamil, in a supervised manner. The morphological generator is an important component of Natural Language Processing in Artificial Intelligence. It generates word forms given a root and affixes. The morphophonemic changes like ad…
▽ More
This paper presents machine learning solutions to a practical problem of Natural Language Generation (NLG), particularly the word formation in agglutinative languages like Tamil, in a supervised manner. The morphological generator is an important component of Natural Language Processing in Artificial Intelligence. It generates word forms given a root and affixes. The morphophonemic changes like addition, deletion, alternation etc., occur when two or more morphemes or words joined together. The Sandhi rules should be explicitly specified in the rule based morphological analyzers and generators. In machine learning framework, these rules can be learned automatically by the system from the training samples and subsequently be applied for new inputs. In this paper we proposed the machine learning models which learn the morphophonemic rules for noun declensions from the given training data. These models are trained to learn sandhi rules using various learning algorithms and the performance of those algorithms are presented. From this we conclude that machine learning of morphological processing such as word form generation can be successfully learned in a supervised manner, without explicit description of rules. The performance of Decision trees and Bayesian machine learning algorithms on noun declensions are discussed.
△ Less
Submitted 14 February, 2014;
originally announced February 2014.
