-
Bridging Structural Dynamics and Biomechanics: Human Motion Estimation through Footstep-Induced Floor Vibrations
Authors:
Yiwen Dong,
Jessica Rose,
Hae Young Noh
Abstract:
Quantitative estimation of human joint motion in daily living spaces is essential for early detection and rehabilitation tracking of neuromusculoskeletal disorders (e.g., Parkinson's) and mitigating trip and fall risks for older adults. Existing approaches involve monitoring devices such as cameras, wearables, and pressure mats, but have operational constraints such as direct line-of-sight, carryi…
▽ More
Quantitative estimation of human joint motion in daily living spaces is essential for early detection and rehabilitation tracking of neuromusculoskeletal disorders (e.g., Parkinson's) and mitigating trip and fall risks for older adults. Existing approaches involve monitoring devices such as cameras, wearables, and pressure mats, but have operational constraints such as direct line-of-sight, carrying devices, and dense deployment. To overcome these limitations, we leverage gait-induced floor vibration to estimate lower-limb joint motion (e.g., ankle, knee, and hip flexion angles), allowing non-intrusive and contactless gait health monitoring in people's living spaces. To overcome the high uncertainty in lower-limb movement given the limited information provided by the gait-induced floor vibrations, we formulate a physics-informed graph to integrate domain knowledge of gait biomechanics and structural dynamics into the model. Specifically, different types of nodes represent heterogeneous information from joint motions and floor vibrations; Their connecting edges represent the physiological relationships between joints and forces governed by gait biomechanics, as well as the relationships between forces and floor responses governed by the structural dynamics. As a result, our model poses physical constraints to reduce uncertainty while allowing information sharing between the body and the floor to make more accurate predictions. We evaluate our approach with 20 participants through a real-world walking experiment. We achieved an average of 3.7 degrees of mean absolute error in estimating 12 joint flexion angles (38% error reduction from baseline), which is comparable to the performance of cameras and wearables in current medical practices.
△ Less
Submitted 21 February, 2025;
originally announced March 2025.
-
Personalized Emotion Detection from Floor Vibrations Induced by Footsteps
Authors:
Yuyan Wu,
Yiwen Dong,
Sumer Vaid,
Gabriella M. Harari,
Hae Young Noh
Abstract:
Emotion recognition is critical for various applications such as early detection of mental health disorders and emotion based smart home systems. Previous studies used various sensing methods for emotion recognition, such as wearable sensors, cameras, and microphones. However, these methods have limitations in long term domestic, including intrusiveness and privacy concerns. To overcome these limi…
▽ More
Emotion recognition is critical for various applications such as early detection of mental health disorders and emotion based smart home systems. Previous studies used various sensing methods for emotion recognition, such as wearable sensors, cameras, and microphones. However, these methods have limitations in long term domestic, including intrusiveness and privacy concerns. To overcome these limitations, this paper introduces a nonintrusive and privacy friendly personalized emotion recognition system, EmotionVibe, which leverages footstep induced floor vibrations for emotion recognition. The main idea of EmotionVibe is that individuals' emotional states influence their gait patterns, subsequently affecting the floor vibrations induced by their footsteps. However, there are two main research challenges: 1) the complex and indirect relationship between human emotions and footstep induced floor vibrations and 2) the large between person variations within the relationship between emotions and gait patterns. To address these challenges, we first empirically characterize this complex relationship and develop an emotion sensitive feature set including gait related and vibration related features from footstep induced floor vibrations. Furthermore, we personalize the emotion recognition system for each user by calculating gait similarities between the target person (i.e., the person whose emotions we aim to recognize) and those in the training dataset and assigning greater weights to training people with similar gait patterns in the loss function. We evaluated our system in a real-world walking experiment with 20 participants, summing up to 37,001 footstep samples. EmotionVibe achieved the mean absolute error (MAE) of 1.11 and 1.07 for valence and arousal score estimations, respectively, reflecting 19.0% and 25.7% error reduction compared to the baseline method.
△ Less
Submitted 6 March, 2025;
originally announced March 2025.
-
Continual Person Identification using Footstep-Induced Floor Vibrations on Heterogeneous Floor Structures
Authors:
Yiwen Dong,
Hae Young Noh
Abstract:
Person identification is important for smart buildings to provide personalized services such as health monitoring, activity tracking, and personnel management. However, previous person identification relies on pre-collected data from everyone, which is impractical in many buildings and public facilities in which visitors are typically expected. This calls for a continual person identification syst…
▽ More
Person identification is important for smart buildings to provide personalized services such as health monitoring, activity tracking, and personnel management. However, previous person identification relies on pre-collected data from everyone, which is impractical in many buildings and public facilities in which visitors are typically expected. This calls for a continual person identification system that gradually learns people's identities on the fly. Existing studies use cameras to achieve this goal, but they require direct line-of-sight and also have raised privacy concerns in public. Other modalities such as wearables and pressure mats are limited by the requirement of device-carrying or dense deployment. Thus, prior studies introduced footstep-induced structural vibration sensing, which is non-intrusive and perceived as more privacy-friendly. However, this approach has a significant challenge: the high variability of vibration data due to structural heterogeneity and human gait variations, which makes online person identification algorithms perform poorly. In this paper, we characterize the variability in footstep-induced structural vibration data for accurate online person identification. To achieve this, we quantify and decompose different sources of variability and then design a feature transformation function to reduce the variability within each person's data to make different people's data more separable. We evaluate our approach through field experiments with 20 people. The results show a 70% variability reduction and a 90% accuracy for online person identification.
△ Less
Submitted 21 February, 2025;
originally announced February 2025.
-
WeVibe: Weight Change Estimation Through Audio-Induced Shelf Vibrations In Autonomous Stores
Authors:
Jiale Zhang,
Yuyan Wu,
Jesse R Codling,
Yen Cheng Chang,
Julia Gersey,
Pei Zhang,
Hae Young Noh,
Yiwen Dong
Abstract:
Weight change estimation is crucial in various applications, particularly for detecting pick-up and put-back actions when people interact with the shelf while shopping in autonomous stores. Moreover, accurate weight change estimation allows autonomous stores to automatically identify items being picked up or put back, ensuring precise cost estimation. However, the conventional approach of estimati…
▽ More
Weight change estimation is crucial in various applications, particularly for detecting pick-up and put-back actions when people interact with the shelf while shopping in autonomous stores. Moreover, accurate weight change estimation allows autonomous stores to automatically identify items being picked up or put back, ensuring precise cost estimation. However, the conventional approach of estimating weight changes requires specialized weight-sensing shelves, which are densely deployed weight scales, incurring intensive sensor consumption and high costs. Prior works explored the vibration-based weight sensing method, but they failed when the location of weight change varies.
In response to these limitations, we made the following contributions: (1) We propose WeVibe, a first item weight change estimation system through active shelf vibration sensing. The main intuition of the system is that the weight placed on the shelf influences the dynamic vibration response of the shelf, thus altering the shelf vibration patterns. (2) We model a physics-informed relationship between the shelf vibration response and item weight across multiple locations on the shelf based on structural dynamics theory. This relationship is linear and allows easy training of a weight estimation model at a new location without heavy data collection. (3) We evaluate our system on a gondola shelf organized as the real-store settings. WeVibe achieved a mean absolute error down to 38.07g and a standard deviation of 31.2g with one sensor and 10% samples from three weight classes on estimating weight change from 0g to 450g, which can be leveraged for differentiating items with more than 100g differences.
△ Less
Submitted 17 February, 2025;
originally announced February 2025.
-
Detecting Gait Abnormalities in Foot-Floor Contacts During Walking Through Footstep-Induced Structural Vibrations
Authors:
Yiwen Dong,
Yuyan Wu,
Hae Young Noh
Abstract:
Gait abnormality detection is critical for the early discovery and progressive tracking of musculoskeletal and neurological disorders, such as Parkinson's and Cerebral Palsy. Especially, analyzing the foot-floor contacts during walking provides important insights into gait patterns, such as contact area, contact force, and contact time, enabling gait abnormality detection through these measurement…
▽ More
Gait abnormality detection is critical for the early discovery and progressive tracking of musculoskeletal and neurological disorders, such as Parkinson's and Cerebral Palsy. Especially, analyzing the foot-floor contacts during walking provides important insights into gait patterns, such as contact area, contact force, and contact time, enabling gait abnormality detection through these measurements. Existing studies use various sensing devices to capture such information, including cameras, wearables, and force plates. However, the former two lack force-related information, making it difficult to identify the causes of gait health issues, while the latter has limited coverage of the walking path. In this study, we leverage footstep-induced structural vibrations to infer foot-floor contact profiles and detect gait abnormalities. The main challenge lies in modeling the complex force transfer mechanism between the foot and the floor surfaces, leading to difficulty in reconstructing the force and contact profile during foot-floor interaction using structural vibrations. To overcome the challenge, we first characterize the floor vibration for each contact type (e.g., heel, midfoot, and toe contact) to understand how contact forces and areas affect the induced floor vibration. Then, we leverage the time-frequency response spectrum resulting from those contacts to develop features that are representative of each contact type. Finally, gait abnormalities are detected by comparing the predicted foot-floor contact force and motion with the healthy gait. To evaluate our approach, we conducted a real-world walking experiment with 8 subjects. Our approach achieves 91.6% and 96.7% accuracy in predicting contact type and time, respectively, leading to 91.9% accuracy in detecting various types of gait abnormalities, including asymmetry, dragging, and midfoot/toe contacts.
△ Less
Submitted 5 August, 2024; v1 submitted 22 May, 2024;
originally announced May 2024.
-
TelecomTM: A Fine-Grained and Ubiquitous Traffic Monitoring System Using Pre-Existing Telecommunication Fiber-Optic Cables as Sensors
Authors:
Jingxiao Liu,
Siyuan Yuan,
Yiwen Dong,
Biondo Biondi,
Hae Young Noh
Abstract:
We introduce the TelecomTM system that uses pre-existing telecommunication fiber-optic cables as virtual strain sensors to sense vehicle-induced ground vibrations for fine-grained and ubiquitous traffic monitoring and characterization. Here we call it a virtual sensor because it is a software-based representation of a physical sensor. Due to the extensively installed telecommunication fiber-optic…
▽ More
We introduce the TelecomTM system that uses pre-existing telecommunication fiber-optic cables as virtual strain sensors to sense vehicle-induced ground vibrations for fine-grained and ubiquitous traffic monitoring and characterization. Here we call it a virtual sensor because it is a software-based representation of a physical sensor. Due to the extensively installed telecommunication fiber-optic cables at the roadside, our system using redundant dark fibers enables to monitor traffic at low cost with low maintenance. Many existing traffic monitoring approaches use cameras, piezoelectric sensors, and smartphones, but they are limited due to privacy concerns and/or deployment requirements. Previous studies attempted to use telecommunication cables for traffic monitoring, but they were only exploratory and limited to simple tasks at a coarse granularity, e.g., vehicle detection, due to their hardware constraints and real-world challenges. In particular, those challenges are 1) unknown and heterogeneous properties of virtual sensors and 2) large and complex noise conditions. To this end, our TelecomTM system first characterizes the geographic location and analyzes the signal pattern of each virtual sensor through driving tests. We then develop a spatial-domain Bayesian filtering and smoothing algorithm to detect, track, and characterize each vehicle. Our approach uses the spatial dependency of multiple virtual sensors and Newton's laws of motion to combine the distributed sensor data to reduce uncertainties in vehicle detection and tracking. In our real-world evaluation on a two-way traffic road with 1120 virtual sensors, TelecomTM achieved 90.18% vehicle detection accuracy, 27$\times$ and 5$\times$ error reduction for vehicle position and speed tracking compared to a baseline method, and $\pm$3.92% and $\pm$11.98% percent error for vehicle wheelbase and weight estimation, respectively.
△ Less
Submitted 4 May, 2023;
originally announced May 2023.
-
Spatial Deep Deconvolution U-Net for Traffic Analyses with Distributed Acoustic Sensing
Authors:
Siyuan Yuan,
Martijn van den Ende,
Jingxiao Liu,
Hae Young Noh,
Robert Clapp,
Cédric Richard,
Biondo Biondi
Abstract:
Distributed Acoustic Sensing (DAS) that transforms city-wide fiber-optic cables into a large-scale strain sensing array has shown the potential to revolutionize urban traffic monitoring by providing a fine-grained, scalable, and low-maintenance monitoring solution. However, the real-world application of DAS is hindered by challenges such as noise contamination and interference among closely travel…
▽ More
Distributed Acoustic Sensing (DAS) that transforms city-wide fiber-optic cables into a large-scale strain sensing array has shown the potential to revolutionize urban traffic monitoring by providing a fine-grained, scalable, and low-maintenance monitoring solution. However, the real-world application of DAS is hindered by challenges such as noise contamination and interference among closely traveling cars. In response, we introduce a self-supervised U-Net model that can suppress background noise and compress car-induced DAS signals into high-resolution pulses through spatial deconvolution. Our work extends recent research by introducing three key advancements. Firstly, we perform a comprehensive resolution analysis of DAS-recorded traffic signals, laying a theoretical foundation for our approach. Secondly, we incorporate space-domain vehicle wavelets into our U-Net model, enabling consistent high-resolution outputs regardless of vehicle speed variations. Finally, we employ L-2 norm regularization in the loss function, enhancing our model's sensitivity to weaker signals from vehicles in remote traffic lanes. We evaluate the effectiveness and robustness of our method through field recordings under different traffic conditions and various driving speeds. Our results show that our method can enhance the spatial-temporal resolution and better resolve closely traveling cars. The spatial deconvolution U-Net model also enables the characterization of large-size vehicles to identify axle numbers and estimate the vehicle length. Monitoring large-size vehicles also benefits imaging deep earth by leveraging the surface waves induced by the dynamic vehicle-road interaction.
△ Less
Submitted 27 June, 2023; v1 submitted 7 December, 2022;
originally announced December 2022.
-
PigV$^2$: Monitoring Pig Vital Signs through Ground Vibrations Induced by Heartbeat and Respiration
Authors:
Yiwen Dong,
Jesse R Codling,
Gary Rohrer,
Jeremy Miles,
Sudhendu Sharma,
Tami Brown-Brandl,
Pei Zhang,
Hae Young Noh
Abstract:
Pig vital sign monitoring (e.g., estimating the heart rate (HR) and respiratory rate (RR)) is essential to understand the stress level of the sow and detect the onset of parturition. It helps to maximize peri-natal survival and improve animal well-being in swine production. The existing approach mainly relies on manual measurement, which is labor-intensive and only provides a few points of informa…
▽ More
Pig vital sign monitoring (e.g., estimating the heart rate (HR) and respiratory rate (RR)) is essential to understand the stress level of the sow and detect the onset of parturition. It helps to maximize peri-natal survival and improve animal well-being in swine production. The existing approach mainly relies on manual measurement, which is labor-intensive and only provides a few points of information. Other sensing modalities such as wearables and cameras are developed to enable more continuous measurement, but are still limited due to animal discomfort, data transfer, and storage challenges. In this paper, we introduce PigV$^2$, the first system to monitor pig heart rate and respiratory rate through ground vibrations. Our approach leverages the insight that both heartbeat and respiration generate ground vibrations when the sow is lying on the floor. We infer vital information by sensing and analyzing these vibrations. The main challenge in developing PigV$^2$ is the overlap of vital- and non-vital-related information in the vibration signals, including pig movements, pig postures, pig-to-sensor distances, and so on. To address this issue, we first characterize their effects, extract their current status, and then reduce their impact by adaptively interpolating vital rates over multiple sensors. PigV$^2$ is evaluated through a real-world deployment with 30 pigs. It has 3.4% and 8.3% average errors in monitoring the HR and RR of the sows, respectively.
△ Less
Submitted 6 December, 2022;
originally announced December 2022.
-
GaitVibe+: Enhancing Structural Vibration-based Footstep Localization Using Temporary Cameras for In-home Gait Analysis
Authors:
Yiwen Dong,
Jingxiao Liu,
Hae Young Noh
Abstract:
In-home gait analysis is important for providing early diagnosis and adaptive treatments for individuals with gait disorders. Existing systems include wearables and pressure mats, but they have limited scalability. Recent studies have developed vision-based systems to enable scalable, accurate in-home gait analysis, but it faces privacy concerns due to the exposure of people's appearances. Our pri…
▽ More
In-home gait analysis is important for providing early diagnosis and adaptive treatments for individuals with gait disorders. Existing systems include wearables and pressure mats, but they have limited scalability. Recent studies have developed vision-based systems to enable scalable, accurate in-home gait analysis, but it faces privacy concerns due to the exposure of people's appearances. Our prior work developed footstep-induced structural vibration sensing for gait monitoring, which is device-free, wide-ranged, and perceived as more privacy-friendly. Although it has succeeded in temporal gait event extraction, it shows limited performance for spatial gait parameter estimation due to imprecise footstep localization. In particular, the localization error mainly comes from the estimation error of the wave arrival time at the vibration sensors and its error propagation to wave velocity estimations. Therefore, we present GaitVibe+, a vibration-based footstep localization method fused with temporarily installed cameras for in-home gait analysis. Our method has two stages: fusion and operating. In the fusion stage, both cameras and vibration sensors are installed to record only a few trials of the subject's footstep data, through which we characterize the uncertainty in wave arrival time and model the wave velocity profiles for the given structure. In the operating stage, we remove the camera to preserve privacy at home. The footstep localization is conducted by estimating the time difference of arrival (TDoA) over multiple vibration sensors, whose accuracy is improved through the reduced uncertainty and velocity modeling during the fusion stage. We evaluate GaitVibe+ through a real-world experiment with 50 walking trials. With only 3 trials of multi-modal fusion, our approach has an average localization error of 0.22 meters, which reduces the spatial gait parameter error from 111% to 27%.
△ Less
Submitted 6 December, 2022;
originally announced December 2022.
-
Vibration-Based Bridge Health Monitoring using Telecommunication Cables
Authors:
Jingxiao Liu,
Siyuan Yuan,
Bin Luo,
Biondo Biondi,
Hae Young Noh
Abstract:
Bridge Health Monitoring (BHM) enables early damage detection of bridges and is thus critical for avoiding more severe damages that might result in major financial and human losses. However, conventional BHM systems require dedicated sensors on bridges, which is costly to install and maintain and hard to scale up. To overcome this challenge, we introduce a new system that uses existing telecommuni…
▽ More
Bridge Health Monitoring (BHM) enables early damage detection of bridges and is thus critical for avoiding more severe damages that might result in major financial and human losses. However, conventional BHM systems require dedicated sensors on bridges, which is costly to install and maintain and hard to scale up. To overcome this challenge, we introduce a new system that uses existing telecommunication cables for Distributed Acoustic Sensing (DAS) to collect bridge dynamic strain responses. In addition, we develop a two-module physics-guided system identification method to extract bridge damage-sensitive information (e.g., natural frequencies and mode shapes) from noisy DAS data by constraining strain and displacement mode shapes by bridge dynamics. This approach does not require installation and maintenance of dedicated sensors on bridges. We evaluate our system with field experiments on a concrete bridge with fiber cable running in a conduit under the deck. Our system successfully identified modal frequencies and reconstructed meter-scale mode shapes.
△ Less
Submitted 10 May, 2022;
originally announced May 2022.
-
Knowledge Transfer between Buildings for Seismic Damage Diagnosis through Adversarial Learning
Authors:
Susu Xu,
Hae Young Noh
Abstract:
Automated structural damage diagnosis after earthquakes is important for improving the efficiency of disaster response and rehabilitation. In conventional data-driven frameworks which use machine learning or statistical models, structural damage diagnosis models are often constructed using supervised learning. The supervised learning requires historical structural response data and corresponding d…
▽ More
Automated structural damage diagnosis after earthquakes is important for improving the efficiency of disaster response and rehabilitation. In conventional data-driven frameworks which use machine learning or statistical models, structural damage diagnosis models are often constructed using supervised learning. The supervised learning requires historical structural response data and corresponding damage states (i.e., labels) for each building to learn the building-specific damage diagnosis model. However, in post-earthquake scenarios, historical data with labels are often not available for many buildings in the affected area. This makes it difficult to construct a damage diagnosis model. Further, directly using the historical data from other buildings to construct a damage diagnosis model for the target building would lead to inaccurate results. This is because each building has unique physical properties and thus unique data distribution. To this end, we introduce a new framework to transfer the model learned from other buildings to diagnose structural damage states in the target building without any labels. This framework is based on an adversarial domain adaptation approach that extracts domain-invariant feature representations of data from different buildings. The feature extraction function is trained in an adversarial way, which ensures that the extracted feature distributions are robust to changes in structures while being predictive of the damage states. With the extracted domain-invariant feature representations, the data distributions become consistent across different buildings. We evaluate our framework on both numerical simulation and field data collected from multiple building structures, which outperforms the state-of-the-art benchmark methods.
△ Less
Submitted 11 October, 2020; v1 submitted 21 February, 2020;
originally announced February 2020.
-
O-MedAL: Online Active Deep Learning for Medical Image Analysis
Authors:
Asim Smailagic,
Pedro Costa,
Alex Gaudio,
Kartik Khandelwal,
Mostafa Mirshekari,
Jonathon Fagert,
Devesh Walawalkar,
Susu Xu,
Adrian Galdran,
Pei Zhang,
Aurélio Campilho,
Hae Young Noh
Abstract:
Active Learning methods create an optimized labeled training set from unlabeled data. We introduce a novel Online Active Deep Learning method for Medical Image Analysis. We extend our MedAL active learning framework to present new results in this paper. Our novel sampling method queries the unlabeled examples that maximize the average distance to all training set examples. Our online method enhanc…
▽ More
Active Learning methods create an optimized labeled training set from unlabeled data. We introduce a novel Online Active Deep Learning method for Medical Image Analysis. We extend our MedAL active learning framework to present new results in this paper. Our novel sampling method queries the unlabeled examples that maximize the average distance to all training set examples. Our online method enhances performance of its underlying baseline deep network. These novelties contribute significant performance improvements, including improving the model's underlying deep network accuracy by 6.30%, using only 25% of the labeled dataset to achieve baseline accuracy, reducing backpropagated images during training by as much as 67%, and demonstrating robustness to class imbalance in binary and multi-class tasks.
△ Less
Submitted 27 July, 2020; v1 submitted 27 August, 2019;
originally announced August 2019.
