-
Fast Bilateral Teleoperation and Imitation Learning Using Sensorless Force Control via Accurate Dynamics Model
Authors:
Koki Yamane,
Yunhan Li,
Masashi Konosu,
Koki Inami,
Junji Oaki,
Sho Sakaino,
Toshiaki Tsuji
Abstract:
In recent years, the advancement of imitation learning has led to increased interest in teleoperating low-cost manipulators to collect demonstration data. However, most existing systems rely on unilateral control, which only transmits target position values. While this approach is easy to implement and suitable for slow, non-contact tasks, it struggles with fast or contact-rich operations due to t…
▽ More
In recent years, the advancement of imitation learning has led to increased interest in teleoperating low-cost manipulators to collect demonstration data. However, most existing systems rely on unilateral control, which only transmits target position values. While this approach is easy to implement and suitable for slow, non-contact tasks, it struggles with fast or contact-rich operations due to the absence of force feedback. This work demonstrates that fast teleoperation with force feedback is feasible even with force-sensorless, low-cost manipulators by leveraging 4-channel bilateral control. Based on accurately identified manipulator dynamics, our method integrates nonlinear terms compensation, velocity and external force estimation, and variable gain corresponding to inertial variation. Furthermore, using data collected by 4-channel bilateral control, we show that incorporating force information into both the input and output of learned policies improves performance in imitation learning. These results highlight the practical effectiveness of our system for high-fidelity teleoperation and data collection on affordable hardware.
△ Less
Submitted 8 July, 2025;
originally announced July 2025.
-
A Survey on Imitation Learning for Contact-Rich Tasks in Robotics
Authors:
Toshiaki Tsuji,
Yasuhiro Kato,
Gokhan Solak,
Heng Zhang,
Tadej Petrič,
Francesco Nori,
Arash Ajoudani
Abstract:
This paper comprehensively surveys research trends in imitation learning for contact-rich robotic tasks. Contact-rich tasks, which require complex physical interactions with the environment, represent a central challenge in robotics due to their nonlinear dynamics and sensitivity to small positional deviations. The paper examines demonstration collection methodologies, including teaching methods a…
▽ More
This paper comprehensively surveys research trends in imitation learning for contact-rich robotic tasks. Contact-rich tasks, which require complex physical interactions with the environment, represent a central challenge in robotics due to their nonlinear dynamics and sensitivity to small positional deviations. The paper examines demonstration collection methodologies, including teaching methods and sensory modalities crucial for capturing subtle interaction dynamics. We then analyze imitation learning approaches, highlighting their applications to contact-rich manipulation. Recent advances in multimodal learning and foundation models have significantly enhanced performance in complex contact tasks across industrial, household, and healthcare domains. Through systematic organization of current research and identification of challenges, this survey provides a foundation for future advancements in contact-rich robotic manipulation.
△ Less
Submitted 16 June, 2025;
originally announced June 2025.
-
Mamba as a motion encoder for robotic imitation learning
Authors:
Toshiaki Tsuji
Abstract:
Recent advancements in imitation learning, particularly with the integration of LLM techniques, are set to significantly improve robots' dexterity and adaptability. This paper proposes using Mamba, a state-of-the-art architecture with potential applications in LLMs, for robotic imitation learning, highlighting its ability to function as an encoder that effectively captures contextual information.…
▽ More
Recent advancements in imitation learning, particularly with the integration of LLM techniques, are set to significantly improve robots' dexterity and adaptability. This paper proposes using Mamba, a state-of-the-art architecture with potential applications in LLMs, for robotic imitation learning, highlighting its ability to function as an encoder that effectively captures contextual information. By reducing the dimensionality of the state space, Mamba operates similarly to an autoencoder. It effectively compresses the sequential information into state variables while preserving the essential temporal dynamics necessary for accurate motion prediction. Experimental results in tasks such as cup placing and case loading demonstrate that despite exhibiting higher estimation errors, Mamba achieves superior success rates compared to Transformers in practical task execution. This performance is attributed to Mamba's structure, which encompasses the state space model. Additionally, the study investigates Mamba's capacity to serve as a real-time motion generator with a limited amount of training data.
△ Less
Submitted 25 September, 2024; v1 submitted 4 September, 2024;
originally announced September 2024.
-
Soft and Rigid Object Grasping With Cross-Structure Hand Using Bilateral Control-Based Imitation Learning
Authors:
Koki Yamane,
Sho Sakaino,
Toshiaki Tsuji
Abstract:
Object grasping is an important ability required for various robot tasks. In particular, tasks that require precise force adjustments during operation, such as grasping an unknown object or using a grasped tool, are difficult for humans to program in advance. Recently, AI-based algorithms that can imitate human force skills have been actively explored as a solution. In particular, bilateral contro…
▽ More
Object grasping is an important ability required for various robot tasks. In particular, tasks that require precise force adjustments during operation, such as grasping an unknown object or using a grasped tool, are difficult for humans to program in advance. Recently, AI-based algorithms that can imitate human force skills have been actively explored as a solution. In particular, bilateral control-based imitation learning achieves human-level motion speeds with environmental adaptability, only requiring human demonstration and without programming. However, owing to hardware limitations, its grasping performance remains limited, and tasks that involves grasping various objects are yet to be achieved. Here, we developed a cross-structure hand to grasp various objects. We experimentally demonstrated that the integration of bilateral control-based imitation learning and the cross-structure hand is effective for grasping various objects and harnessing tools.
△ Less
Submitted 15 November, 2023;
originally announced November 2023.
-
Assessing the optimal contributions of renewables and carbon capture and storage toward carbon neutrality by 2050
Authors:
Dinh Hoa Nguyen,
Andrew Chapman,
Takeshi Tsuji
Abstract:
Building on the carbon reduction targets agreed in the Paris Agreements, many nations have renewed their efforts toward achieving carbon neutrality by the year 2050. In line with this ambitious goal, nations are seeking to understand the appropriate combination of technologies which will enable the required reductions in such a way that they are appealing to investors. Around the globe, solar and…
▽ More
Building on the carbon reduction targets agreed in the Paris Agreements, many nations have renewed their efforts toward achieving carbon neutrality by the year 2050. In line with this ambitious goal, nations are seeking to understand the appropriate combination of technologies which will enable the required reductions in such a way that they are appealing to investors. Around the globe, solar and wind power lead in terms of renewable energy deployment, while carbon capture and storage (CCS) is scaling up toward making a significant contribution to deep carbon cuts.
Using Japan as a case study nation, this research proposes a linear optimization modeling approach to identify the potential contributions of renewables and CCS toward maximizing carbon reduction and identifying their economic merits over time. Results identify that the combination of these three technologies could enable a carbon dioxide emission reduction of between 55 and 67 percent in the energy sector by 2050 depending on resilience levels and CCS deployment regimes. Further reductions are likely to emerge with increased carbon pricing over time.
△ Less
Submitted 9 May, 2023;
originally announced May 2023.
-
Automated Classification of General Movements in Infants Using a Two-stream Spatiotemporal Fusion Network
Authors:
Yuki Hashimoto,
Akira Furui,
Koji Shimatani,
Maura Casadio,
Paolo Moretti,
Pietro Morasso,
Toshio Tsuji
Abstract:
The assessment of general movements (GMs) in infants is a useful tool in the early diagnosis of neurodevelopmental disorders. However, its evaluation in clinical practice relies on visual inspection by experts, and an automated solution is eagerly awaited. Recently, video-based GMs classification has attracted attention, but this approach would be strongly affected by irrelevant information, such…
▽ More
The assessment of general movements (GMs) in infants is a useful tool in the early diagnosis of neurodevelopmental disorders. However, its evaluation in clinical practice relies on visual inspection by experts, and an automated solution is eagerly awaited. Recently, video-based GMs classification has attracted attention, but this approach would be strongly affected by irrelevant information, such as background clutter in the video. Furthermore, for reliability, it is necessary to properly extract the spatiotemporal features of infants during GMs. In this study, we propose an automated GMs classification method, which consists of preprocessing networks that remove unnecessary background information from GMs videos and adjust the infant's body position, and a subsequent motion classification network based on a two-stream structure. The proposed method can efficiently extract the essential spatiotemporal features for GMs classification while preventing overfitting to irrelevant information for different recording environments. We validated the proposed method using videos obtained from 100 infants. The experimental results demonstrate that the proposed method outperforms several baseline models and the existing methods.
△ Less
Submitted 4 July, 2022;
originally announced July 2022.
-
Force control of grinding process based on frequency analysis
Authors:
Yuya Nogi,
Sho Sakaino,
Toshiaki Tsuji
Abstract:
Hysteresis-induced drift is a major issue in the detection of force induced during grinding and cutting operations. In this paper, we propose an external force estimation method based on the Mel spectrogram of the force obtained from a force sensor. We focus on the frequent strong correlation between the vibration frequency and the external force in operations with periodic vibrations. The frequen…
▽ More
Hysteresis-induced drift is a major issue in the detection of force induced during grinding and cutting operations. In this paper, we propose an external force estimation method based on the Mel spectrogram of the force obtained from a force sensor. We focus on the frequent strong correlation between the vibration frequency and the external force in operations with periodic vibrations. The frequency information is found to be more effective for an accurate force estimation than the amplitude in cases with large noise caused by vibration. We experimentally demonstrate that the force estimation method that combines the Mel spectrogram with a neural network is robust against drift.
△ Less
Submitted 7 December, 2021; v1 submitted 6 December, 2021;
originally announced December 2021.
-
Generation Drawing/Grinding Trajectoy Based on Hierarchical CVAE
Authors:
Masahiro Aita,
Keito Sugawara,
Sho Sakaino,
Toshiaki Tsuji
Abstract:
In this study, we propose a method to model the local and global features of the drawing/grinding trajectory with hierarchical Variational Autoencoders (VAEs). By combining two separately trained VAE models in a hierarchical structure, it is possible to generate trajectories with high reproducibility for both local and global features. The hierarchical generation network enables the generation of…
▽ More
In this study, we propose a method to model the local and global features of the drawing/grinding trajectory with hierarchical Variational Autoencoders (VAEs). By combining two separately trained VAE models in a hierarchical structure, it is possible to generate trajectories with high reproducibility for both local and global features. The hierarchical generation network enables the generation of higher-order trajectories with a relatively small amount of training data. The simulation and experimental results demonstrate the generalization performance of the proposed method. In addition, we confirmed that it is possible to generate new trajectories, which have never been learned in the past, by changing the combination of the learned models.
△ Less
Submitted 23 November, 2021; v1 submitted 21 November, 2021;
originally announced November 2021.
-
A Time-Series Scale Mixture Model of EEG with a Hidden Markov Structure for Epileptic Seizure Detection
Authors:
Akira Furui,
Tomoyuki Akiyama,
Toshio Tsuji
Abstract:
In this paper, we propose a time-series stochastic model based on a scale mixture distribution with Markov transitions to detect epileptic seizures in electroencephalography (EEG). In the proposed model, an EEG signal at each time point is assumed to be a random variable following a Gaussian distribution. The covariance matrix of the Gaussian distribution is weighted with a latent scale parameter,…
▽ More
In this paper, we propose a time-series stochastic model based on a scale mixture distribution with Markov transitions to detect epileptic seizures in electroencephalography (EEG). In the proposed model, an EEG signal at each time point is assumed to be a random variable following a Gaussian distribution. The covariance matrix of the Gaussian distribution is weighted with a latent scale parameter, which is also a random variable, resulting in the stochastic fluctuations of covariances. By introducing a latent state variable with a Markov chain in the background of this stochastic relationship, time-series changes in the distribution of latent scale parameters can be represented according to the state of epileptic seizures. In an experiment, we evaluated the performance of the proposed model for seizure detection using EEGs with multiple frequency bands decomposed from a clinical dataset. The results demonstrated that the proposed model can detect seizures with high sensitivity and outperformed several baselines.
△ Less
Submitted 11 November, 2021;
originally announced November 2021.
-
EMG Pattern Recognition via Bayesian Inference with Scale Mixture-Based Stochastic Generative Models
Authors:
Akira Furui,
Takuya Igaue,
Toshio Tsuji
Abstract:
Electromyogram (EMG) has been utilized to interface signals for prosthetic hands and information devices owing to its ability to reflect human motion intentions. Although various EMG classification methods have been introduced into EMG-based control systems, they do not fully consider the stochastic characteristics of EMG signals. This paper proposes an EMG pattern classification method incorporat…
▽ More
Electromyogram (EMG) has been utilized to interface signals for prosthetic hands and information devices owing to its ability to reflect human motion intentions. Although various EMG classification methods have been introduced into EMG-based control systems, they do not fully consider the stochastic characteristics of EMG signals. This paper proposes an EMG pattern classification method incorporating a scale mixture-based generative model. A scale mixture model is a stochastic EMG model in which the EMG variance is considered as a random variable, enabling the representation of uncertainty in the variance. This model is extended in this study and utilized for EMG pattern classification. The proposed method is trained by variational Bayesian learning, thereby allowing the automatic determination of the model complexity. Furthermore, to optimize the hyperparameters of the proposed method with a partial discriminative approach, a mutual information-based determination method is introduced. Simulation and EMG analysis experiments demonstrated the relationship between the hyperparameters and classification accuracy of the proposed method as well as the validity of the proposed method. The comparison using public EMG datasets revealed that the proposed method outperformed the various conventional classifiers. These results indicated the validity of the proposed method and its applicability to EMG-based control systems. In EMG pattern recognition, a classifier based on a generative model that reflects the stochastic characteristics of EMG signals can outperform the conventional general-purpose classifier.
△ Less
Submitted 20 July, 2021;
originally announced July 2021.
-
Biomimetic Control of Myoelectric Prosthetic Hand Based on a Lambda-type Muscle Model
Authors:
Akira Furui,
Kosuke Nakagaki,
Toshio Tsuji
Abstract:
Myoelectric prosthetic hands are intended to replace the function of the amputee's lost arm. Therefore, developing robotic prosthetics that can mimic not only the appearance and functionality of humans but also characteristics unique to human movements is paramount. Although the impedance model was proposed to realize biomimetic control, this model cannot replicate the characteristics of human mov…
▽ More
Myoelectric prosthetic hands are intended to replace the function of the amputee's lost arm. Therefore, developing robotic prosthetics that can mimic not only the appearance and functionality of humans but also characteristics unique to human movements is paramount. Although the impedance model was proposed to realize biomimetic control, this model cannot replicate the characteristics of human movements effectively because the joint angle always converges to the equilibrium position during muscle relaxation. This paper proposes a novel biomimetic control method for myoelectric prosthetic hands integrating the impedance model with the concept of the $λ$-type muscle model. The proposed method can dynamically control the joint equilibrium position, according to the state of the muscle, and can maintain the joint angle naturally during muscle relaxation. The effectiveness of the proposed method is evaluated through simulations and a series of experiments on non-amputee participants. The experimental results, based on comparison with the actual human joint angles, suggest that the proposed method has a better correlation with the actual human motion than the conventional methods. Additionally, the control experiments showed that the proposed method could achieve a natural prosthetic hand movement similar to that of a human, thereby allowing voluntary hand opening and closing movements.
△ Less
Submitted 29 May, 2021;
originally announced May 2021.
-
Imitation Learning for Variable Speed Contact Motion for Operation up to Control Bandwidth
Authors:
Sho Sakaino,
Kazuki Fujimoto,
Yuki Saigusa,
Toshiaki Tsuji
Abstract:
The generation of robot motions in the real world is difficult by using conventional controllers alone and requires highly intelligent processing. In this regard, learning-based motion generations are currently being investigated. However, the main issue has been improvements of the adaptability to spatially varying environments, but a variation of the operating speed has not been investigated in…
▽ More
The generation of robot motions in the real world is difficult by using conventional controllers alone and requires highly intelligent processing. In this regard, learning-based motion generations are currently being investigated. However, the main issue has been improvements of the adaptability to spatially varying environments, but a variation of the operating speed has not been investigated in detail. In contact-rich tasks, it is especially important to be able to adjust the operating speed because a nonlinear relationship occurs between the operating speed and force (e.g., inertial and frictional forces), and it affects the results of the tasks. Therefore, in this study, we propose a method for generating variable operating speeds while adapting to spatial perturbations in the environment. The proposed method can be adapted to nonlinearities by utilizing a small amount of motion data. We experimentally evaluated the proposed method by erasing a line using an eraser fixed to the tip of the robot as an example of a contact-rich task. Furthermore, the proposed method enables a robot to perform a task faster than a human operator and is capable of operating close to the control bandwidth.
△ Less
Submitted 12 February, 2022; v1 submitted 20 February, 2021;
originally announced February 2021.
-
Non-Gaussianity Detection of EEG Signals Based on a Multivariate Scale Mixture Model for Diagnosis of Epileptic Seizures
Authors:
Akira Furui,
Ryota Onishi,
Akihito Takeuchi,
Tomoyuki Akiyama,
Toshio Tsuji
Abstract:
Objective: The detection of epileptic seizures from scalp electroencephalogram (EEG) signals can facilitate early diagnosis and treatment. Previous studies suggested that the Gaussianity of EEG distributions changes depending on the presence or absence of seizures; however, no general EEG signal models can explain such changes in distributions within a unified scheme. Methods: This paper describes…
▽ More
Objective: The detection of epileptic seizures from scalp electroencephalogram (EEG) signals can facilitate early diagnosis and treatment. Previous studies suggested that the Gaussianity of EEG distributions changes depending on the presence or absence of seizures; however, no general EEG signal models can explain such changes in distributions within a unified scheme. Methods: This paper describes the formulation of a stochastic EEG model based on a multivariate scale mixture distribution that can represent changes in non-Gaussianity caused by stochastic fluctuations in EEG. In addition, we propose an EEG analysis method by combining the model with a filter bank and introduce a feature representing the non-Gaussianity latent in each EEG frequency band. Results: We applied the proposed method to multichannel EEG data from twenty patients with focal epilepsy. The results showed a significant increase in the proposed feature during epileptic seizures, particularly in the high-frequency band. The feature calculated in the high-frequency band allowed highly accurate classification of seizure and non-seizure segments [area under the receiver operating characteristic curve (AUC) = 0.881] using only a simple threshold. Conclusion: This paper proposed a multivariate scale mixture distribution-based stochastic EEG model capable of representing non-Gaussianity associated with epileptic seizures. Experiments using simulated and real EEG data demonstrated the validity of the model and its applicability to epileptic seizure detection. Significance: The stochastic fluctuations of EEG quantified by the proposed model can help detect epileptic seizures with high accuracy.
△ Less
Submitted 2 July, 2020;
originally announced July 2020.
-
Assembly robots with optimized control stiffness through reinforcement learning
Authors:
Masahide Oikawa,
Kyo Kutsuzawa,
Sho Sakaino,
Toshiaki Tsuji
Abstract:
There is an increased demand for task automation in robots. Contact-rich tasks, wherein multiple contact transitions occur in a series of operations, are extensively being studied to realize high accuracy. In this study, we propose a methodology that uses reinforcement learning (RL) to achieve high performance in robots for the execution of assembly tasks that require precise contact with objects…
▽ More
There is an increased demand for task automation in robots. Contact-rich tasks, wherein multiple contact transitions occur in a series of operations, are extensively being studied to realize high accuracy. In this study, we propose a methodology that uses reinforcement learning (RL) to achieve high performance in robots for the execution of assembly tasks that require precise contact with objects without causing damage. The proposed method ensures the online generation of stiffness matrices that help improve the performance of local trajectory optimization. The method has an advantage of rapid response owing to short sampling time of the trajectory planning. The effectiveness of the method was verified via experiments involving two contact-rich tasks. The results indicate that the proposed method can be implemented in various contact-rich manipulations. A demonstration video shows the performance. (https://youtu.be/gxSCl7Tp4-0)
△ Less
Submitted 27 February, 2020;
originally announced February 2020.
-
A Scale Mixture-Based Stochastic Model of Surface EMG Signals With Variable Variances
Authors:
Akira Furui,
Hideaki Hayashi,
Toshio Tsuji
Abstract:
Objective: Surface electromyogram (EMG) signals have typically been assumed to follow a Gaussian distribution. However, the presence of non-Gaussian signals associated with muscle activity has been reported in recent studies, and there is no general model of the distribution of EMG signals that can explain both non-Gaussian and Gaussian distributions within a unified scheme. Methods: In this paper…
▽ More
Objective: Surface electromyogram (EMG) signals have typically been assumed to follow a Gaussian distribution. However, the presence of non-Gaussian signals associated with muscle activity has been reported in recent studies, and there is no general model of the distribution of EMG signals that can explain both non-Gaussian and Gaussian distributions within a unified scheme. Methods: In this paper, we describe the formulation of a non-Gaussian EMG model based on a scale mixture distribution. In the model, an EMG signal at a certain time follows a Gaussian distribution, and its variance is handled as a random variable that follows an inverse gamma distribution. Accordingly, the probability distribution of EMG signals is assumed to be a mixture of Gaussians with the same mean but different variances. The EMG variance distribution is estimated via marginal likelihood maximization. Results: Experiments involving nine participants revealed that the proposed model provides a better fit to recorded EMG signals than conventional EMG models. It was also shown that variance distribution parameters may reflect underlying motor unit activity. Conclusion: This study proposed a scale mixture distribution-based stochastic EMG model capable of representing changes in non-Gaussianity associated with muscle activity. A series of experiments demonstrated the validity of the model and highlighted the relationship between the variance distribution and muscle force. Significance: The proposed model helps to clarify conventional wisdom regarding the probability distribution of surface EMG signals within a unified scheme.
△ Less
Submitted 10 December, 2019;
originally announced December 2019.
-
A Neural Network Based on the Johnson $S_\mathrm{U}$ Translation System and Related Application to Electromyogram Classification
Authors:
Hideaki Hayashi,
Taro Shibanoki,
Toshio Tsuji
Abstract:
Electromyogram (EMG) classification is a key technique in EMG-based control systems. The existing EMG classification methods do not consider the characteristics of EMG features that the distribution has skewness and kurtosis, causing drawbacks such as the requirement of hyperparameter tuning. In this paper, we propose a neural network based on the Johnson $S_\mathrm{U}$ translation system that is…
▽ More
Electromyogram (EMG) classification is a key technique in EMG-based control systems. The existing EMG classification methods do not consider the characteristics of EMG features that the distribution has skewness and kurtosis, causing drawbacks such as the requirement of hyperparameter tuning. In this paper, we propose a neural network based on the Johnson $S_\mathrm{U}$ translation system that is capable of representing distributions with skewness and kurtosis. The Johnson system is a normalizing translation that transforms non-normal data to a normal distribution, thereby enabling the representation of a wide range of distributions. In this study, a discriminative model based on the multivariate Johnson $S_\mathrm{U}$ translation system is transformed into a linear combination of coefficients and input vectors using log-linearization. This is then incorporated into a neural network structure, thereby allowing the calculation of the posterior probability of the input vectors for each class and the determination of model parameters as weight coefficients of the network. The uniqueness of convergence of the network learning is theoretically guaranteed. In the experiments, the suitability of the proposed network for distributions including skewness and kurtosis is evaluated using artificially generated data. Its applicability for real biological data is also evaluated via an EMG classification experiment. The results show that the proposed network achieves high classification performance without the need for hyperparameter optimization.
△ Less
Submitted 14 November, 2019;
originally announced December 2019.
-
Design of Resonance Ratio Control with Relative Position Information for Two-inertia System
Authors:
Kenta Araake,
Sho Sakaino,
Toshiaki Tsuji
Abstract:
Two-inertia systems are prone to resonance vibrations that degrade their control performances. These unwanted vibrations can be effectively suppressed by control methods based on a disturbance observer (DOB). Vibration suppression control methods using the information of both the motor and load sides have been widely researched in recent years. Methods that exploit the spring deflection or torsion…
▽ More
Two-inertia systems are prone to resonance vibrations that degrade their control performances. These unwanted vibrations can be effectively suppressed by control methods based on a disturbance observer (DOB). Vibration suppression control methods using the information of both the motor and load sides have been widely researched in recent years. Methods that exploit the spring deflection or torsional force of two-inertia systems have delivered promising performances. However, few conventional methods have exploited the relative position information, and the discussion of position control is currently insufficient. Focusing on the relative position, this study proposes a new resonance ratio control (RRC) based on the relative acceleration and state feedback. The structure of the proposed RRC is derived theoretically and the proposed method is experimentally validated.
△ Less
Submitted 19 September, 2019;
originally announced September 2019.
