-
Meta-Learning Online Dynamics Model Adaptation in Off-Road Autonomous Driving
Authors:
Jacob Levy,
Jason Gibson,
Bogdan Vlahov,
Erica Tevere,
Evangelos Theodorou,
David Fridovich-Keil,
Patrick Spieler
Abstract:
High-speed off-road autonomous driving presents unique challenges due to complex, evolving terrain characteristics and the difficulty of accurately modeling terrain-vehicle interactions. While dynamics models used in model-based control can be learned from real-world data, they often struggle to generalize to unseen terrain, making real-time adaptation essential. We propose a novel framework that…
▽ More
High-speed off-road autonomous driving presents unique challenges due to complex, evolving terrain characteristics and the difficulty of accurately modeling terrain-vehicle interactions. While dynamics models used in model-based control can be learned from real-world data, they often struggle to generalize to unseen terrain, making real-time adaptation essential. We propose a novel framework that combines a Kalman filter-based online adaptation scheme with meta-learned parameters to address these challenges. Offline meta-learning optimizes the basis functions along which adaptation occurs, as well as the adaptation parameters, while online adaptation dynamically adjusts the onboard dynamics model in real time for model-based control. We validate our approach through extensive experiments, including real-world testing on a full-scale autonomous off-road vehicle, demonstrating that our method outperforms baseline approaches in prediction accuracy, performance, and safety metrics, particularly in safety-critical scenarios. Our results underscore the effectiveness of meta-learned dynamics model adaptation, advancing the development of reliable autonomous systems capable of navigating diverse and unseen environments. Video is available at: https://youtu.be/cCKHHrDRQEA
△ Less
Submitted 23 April, 2025;
originally announced April 2025.
-
Brain-to-Text Decoding: A Non-invasive Approach via Typing
Authors:
Jarod Lévy,
Mingfang Zhang,
Svetlana Pinet,
Jérémy Rapin,
Hubert Banville,
Stéphane d'Ascoli,
Jean-Rémi King
Abstract:
Modern neuroprostheses can now restore communication in patients who have lost the ability to speak or move. However, these invasive devices entail risks inherent to neurosurgery. Here, we introduce a non-invasive method to decode the production of sentences from brain activity and demonstrate its efficacy in a cohort of 35 healthy volunteers. For this, we present Brain2Qwerty, a new deep learning…
▽ More
Modern neuroprostheses can now restore communication in patients who have lost the ability to speak or move. However, these invasive devices entail risks inherent to neurosurgery. Here, we introduce a non-invasive method to decode the production of sentences from brain activity and demonstrate its efficacy in a cohort of 35 healthy volunteers. For this, we present Brain2Qwerty, a new deep learning architecture trained to decode sentences from either electro- (EEG) or magneto-encephalography (MEG), while participants typed briefly memorized sentences on a QWERTY keyboard. With MEG, Brain2Qwerty reaches, on average, a character-error-rate (CER) of 32% and substantially outperforms EEG (CER: 67%). For the best participants, the model achieves a CER of 19%, and can perfectly decode a variety of sentences outside of the training set. While error analyses suggest that decoding depends on motor processes, the analysis of typographical errors suggests that it also involves higher-level cognitive factors. Overall, these results narrow the gap between invasive and non-invasive methods and thus open the path for developing safe brain-computer interfaces for non-communicating patients.
△ Less
Submitted 18 February, 2025;
originally announced February 2025.
-
Active Inverse Learning in Stackelberg Trajectory Games
Authors:
William Ward,
Yue Yu,
Jacob Levy,
Negar Mehr,
David Fridovich-Keil,
Ufuk Topcu
Abstract:
Game-theoretic inverse learning is the problem of inferring a player's objectives from their actions. We formulate an inverse learning problem in a Stackelberg game between a leader and a follower, where each player's action is the trajectory of a dynamical system. We propose an active inverse learning method for the leader to infer which hypothesis among a finite set of candidates best describes…
▽ More
Game-theoretic inverse learning is the problem of inferring a player's objectives from their actions. We formulate an inverse learning problem in a Stackelberg game between a leader and a follower, where each player's action is the trajectory of a dynamical system. We propose an active inverse learning method for the leader to infer which hypothesis among a finite set of candidates best describes the follower's objective function. Instead of using passively observed trajectories like existing methods, we actively maximize the differences in the follower's trajectories under different hypotheses by optimizing the leader's control inputs. Compared with uniformly random inputs, the optimized inputs accelerate the convergence of the estimated probability of different hypotheses conditioned on the follower's trajectory. We demonstrate the proposed method in a receding-horizon repeated trajectory game and simulate the results using virtual TurtleBots in Gazebo.
△ Less
Submitted 11 October, 2024; v1 submitted 15 August, 2023;
originally announced August 2023.
-
Simultaneous column-based deep learning progression analysis of atrophy associated with AMD in longitudinal OCT studies
Authors:
Adi Szeskin,
Roei Yehuda,
Or Shmueli,
Jaime Levy,
Leo Joskowicz
Abstract:
Purpose: Disease progression of retinal atrophy associated with AMD requires the accurate quantification of the retinal atrophy changes on longitudinal OCT studies. It is based on finding, comparing, and delineating subtle atrophy changes on consecutive pairs (prior and current) of unregistered OCT scans. Methods: We present a fully automatic end-to-end pipeline for the simultaneous detection and…
▽ More
Purpose: Disease progression of retinal atrophy associated with AMD requires the accurate quantification of the retinal atrophy changes on longitudinal OCT studies. It is based on finding, comparing, and delineating subtle atrophy changes on consecutive pairs (prior and current) of unregistered OCT scans. Methods: We present a fully automatic end-to-end pipeline for the simultaneous detection and quantification of time-related atrophy changes associated with dry AMD in pairs of OCT scans of a patient. It uses a novel simultaneous multi-channel column-based deep learning model trained on registered pairs of OCT scans that concurrently detects and segments retinal atrophy segments in consecutive OCT scans by classifying light scattering patterns in matched pairs of vertical pixel-wide columns (A-scans) in registered prior and current OCT slices (B-scans). Results: Experimental results on 4,040 OCT slices with 5.2M columns from 40 scans pairs of 18 patients (66% training/validation, 33% testing) with 24.13+-14.0 months apart in which Complete RPE and Outer Retinal Atrophy (cRORA) was identified in 1,998 OCT slices (735 atrophy lesions from 3,732 segments, 0.45M columns) yield a mean atrophy segments detection precision, recall of 0.90+-0.09, 0.95+-0.06 and 0.74+-0.18, 0.94+-0.12 for atrophy lesions with AUC=0.897, all above observer variability. Simultaneous classification outperforms standalone classification precision and recall by 30+-62% and 27+-0% for atrophy segments and lesions. Conclusions: simultaneous column-based detection and quantification of retinal atrophy changes associated with AMD is accurate and outperforms standalone classification methods. Translational relevance: an automatic and efficient way to detect and quantify retinal atrophy changes associated with AMD.
△ Less
Submitted 31 July, 2023;
originally announced July 2023.
-
Distortion measure of spectrograms for classification of respiratory diseases
Authors:
Jeremy Levy,
Alexander Naitsat,
Yehoshua Y. Zeevi
Abstract:
A new method for the classification of respiratory diseases is presented. The method is based on a novel class of features, extracted from pulmonary sounds, by parameterizing their spectrograms that are represented as surfaces, and by utilizing geometrical distortions defined with reference to these surfaces. This method yields a set of highly descriptive features for the analysis of pulmonary sou…
▽ More
A new method for the classification of respiratory diseases is presented. The method is based on a novel class of features, extracted from pulmonary sounds, by parameterizing their spectrograms that are represented as surfaces, and by utilizing geometrical distortions defined with reference to these surfaces. This method yields a set of highly descriptive features for the analysis of pulmonary sound recordings. Furthermore, by combining these features with Mel-frequency cepstral coefficients, we introduce a powerful model for the automatic diagnosis of common respiratory pathologies. Compared with baseline methods, our model achieves superior results for binary and multi-class classifications of common respiratory diseases. Our new approach to the classification of one-dimensional signals is applicable to other signals in the context of their representations in combined spaces or manifolds.
△ Less
Submitted 4 June, 2021;
originally announced June 2021.
-
Contrastive Unsupervised Learning for Speech Emotion Recognition
Authors:
Mao Li,
Bo Yang,
Joshua Levy,
Andreas Stolcke,
Viktor Rozgic,
Spyros Matsoukas,
Constantinos Papayiannis,
Daniel Bone,
Chao Wang
Abstract:
Speech emotion recognition (SER) is a key technology to enable more natural human-machine communication. However, SER has long suffered from a lack of public large-scale labeled datasets. To circumvent this problem, we investigate how unsupervised representation learning on unlabeled datasets can benefit SER. We show that the contrastive predictive coding (CPC) method can learn salient representat…
▽ More
Speech emotion recognition (SER) is a key technology to enable more natural human-machine communication. However, SER has long suffered from a lack of public large-scale labeled datasets. To circumvent this problem, we investigate how unsupervised representation learning on unlabeled datasets can benefit SER. We show that the contrastive predictive coding (CPC) method can learn salient representations from unlabeled datasets, which improves emotion recognition performance. In our experiments, this method achieved state-of-the-art concordance correlation coefficient (CCC) performance for all emotion primitives (activation, valence, and dominance) on IEMOCAP. Additionally, on the MSP- Podcast dataset, our method obtained considerable performance improvements compared to baselines.
△ Less
Submitted 12 February, 2021;
originally announced February 2021.
-
Machine learning for nocturnal diagnosis of chronic obstructive pulmonary disease using digital oximetry biomarkers
Authors:
Jeremy Levy,
Daniel Alvarez,
Felix del Campo,
Joachim A. Behar
Abstract:
Objective: Chronic obstructive pulmonary disease (COPD) is a highly prevalent chronic condition. COPD is a major source of morbidity, mortality and healthcare costs. Spirometry is the gold standard test for a definitive diagnosis and severity grading of COPD. However, a large proportion of individuals with COPD are undiagnosed and untreated. Given the high prevalence of COPD and its clinical impor…
▽ More
Objective: Chronic obstructive pulmonary disease (COPD) is a highly prevalent chronic condition. COPD is a major source of morbidity, mortality and healthcare costs. Spirometry is the gold standard test for a definitive diagnosis and severity grading of COPD. However, a large proportion of individuals with COPD are undiagnosed and untreated. Given the high prevalence of COPD and its clinical importance, it is critical to develop new algorithms to identify undiagnosed COPD, especially in specific groups at risk, such as those with sleep disorder breathing. To our knowledge, no research has looked at the feasibility of COPD diagnosis from the nocturnal oximetry time series. Approach: We hypothesize that patients with COPD will exert certain patterns and/or dynamics of their overnight oximetry time series that are unique to this condition. We introduce a novel approach to nocturnal COPD diagnosis using 44 oximetry digital biomarkers and 5 demographic features and assess its performance in a population sample at risk of sleep-disordered breathing. A total of n=350 unique patients polysomnography (PSG) recordings. A random forest (RF) classifier is trained using these features and evaluated using the nested cross-validation procedure. Significance: Our research makes a number of novel scientific contributions. First, we demonstrated for the first time, the feasibility of COPD diagnosis from nocturnal oximetry time series in a population sample at risk of sleep disordered breathing. We highlighted what digital oximetry biomarkers best reflect how COPD manifests overnight. The results motivate that overnight single channel oximetry is a valuable pathway for COPD diagnosis.
△ Less
Submitted 10 December, 2020;
originally announced December 2020.
-
Speech Sentiment and Customer Satisfaction Estimation in Socialbot Conversations
Authors:
Yelin Kim,
Joshua Levy,
Yang Liu
Abstract:
For an interactive agent, such as task-oriented spoken dialog systems or chatbots, measuring and adapting to Customer Satisfaction (CSAT) is critical in order to understand user perception of an agent's behavior and increase user engagement and retention. However, an agent often relies on explicit customer feedback for measuring CSAT. Such explicit feedback may result in potential distraction to u…
▽ More
For an interactive agent, such as task-oriented spoken dialog systems or chatbots, measuring and adapting to Customer Satisfaction (CSAT) is critical in order to understand user perception of an agent's behavior and increase user engagement and retention. However, an agent often relies on explicit customer feedback for measuring CSAT. Such explicit feedback may result in potential distraction to users and it can be challenging to capture continuously changing user's satisfaction. To address this challenge, we present a new approach to automatically estimate CSAT using acoustic and lexical information in the Alexa Prize Socialbot data. We first explore the relationship between CSAT and sentiment scores at both the utterance and conversation level. We then investigate static and temporal modeling methods that use estimated sentiment scores as a mid-level representation. The results show that the sentiment scores, particularly valence and satisfaction, are correlated with CSAT. We also demonstrate that our proposed temporal modeling approach for estimating CSAT achieves competitive performance, relative to static baselines as well as human performance. This work provides insights into open domain social conversations between real users and socialbots, and the use of both acoustic and lexical information for understanding the relationship between CSAT and sentiment scores.
△ Less
Submitted 27 August, 2020;
originally announced August 2020.
