-
ExoStart: Efficient learning for dexterous manipulation with sensorized exoskeleton demonstrations
Authors:
Zilin Si,
Jose Enrique Chen,
M. Emre Karagozler,
Antonia Bronars,
Jonathan Hutchinson,
Thomas Lampe,
Nimrod Gileadi,
Taylor Howell,
Stefano Saliceti,
Lukasz Barczyk,
Ilan Olivarez Correa,
Tom Erez,
Mohit Shridhar,
Murilo Fernandes Martins,
Konstantinos Bousmalis,
Nicolas Heess,
Francesco Nori,
Maria Bauza Villalonga
Abstract:
Recent advancements in teleoperation systems have enabled high-quality data collection for robotic manipulators, showing impressive results in learning manipulation at scale. This progress suggests that extending these capabilities to robotic hands could unlock an even broader range of manipulation skills, especially if we could achieve the same level of dexterity that human hands exhibit. However…
▽ More
Recent advancements in teleoperation systems have enabled high-quality data collection for robotic manipulators, showing impressive results in learning manipulation at scale. This progress suggests that extending these capabilities to robotic hands could unlock an even broader range of manipulation skills, especially if we could achieve the same level of dexterity that human hands exhibit. However, teleoperating robotic hands is far from a solved problem, as it presents a significant challenge due to the high degrees of freedom of robotic hands and the complex dynamics occurring during contact-rich settings. In this work, we present ExoStart, a general and scalable learning framework that leverages human dexterity to improve robotic hand control. In particular, we obtain high-quality data by collecting direct demonstrations without a robot in the loop using a sensorized low-cost wearable exoskeleton, capturing the rich behaviors that humans can demonstrate with their own hands. We also propose a simulation-based dynamics filter that generates dynamically feasible trajectories from the collected demonstrations and use the generated trajectories to bootstrap an auto-curriculum reinforcement learning method that relies only on simple sparse rewards. The ExoStart pipeline is generalizable and yields robust policies that transfer zero-shot to the real robot. Our results demonstrate that ExoStart can generate dexterous real-world hand skills, achieving a success rate above 50% on a wide range of complex tasks such as opening an AirPods case or inserting and turning a key in a lock. More details and videos can be found in https://sites.google.com/view/exostart.
△ Less
Submitted 13 June, 2025;
originally announced June 2025.
-
DemoStart: Demonstration-led auto-curriculum applied to sim-to-real with multi-fingered robots
Authors:
Maria Bauza,
Jose Enrique Chen,
Valentin Dalibard,
Nimrod Gileadi,
Roland Hafner,
Murilo F. Martins,
Joss Moore,
Rugile Pevceviciute,
Antoine Laurens,
Dushyant Rao,
Martina Zambelli,
Martin Riedmiller,
Jon Scholz,
Konstantinos Bousmalis,
Francesco Nori,
Nicolas Heess
Abstract:
We present DemoStart, a novel auto-curriculum reinforcement learning method capable of learning complex manipulation behaviors on an arm equipped with a three-fingered robotic hand, from only a sparse reward and a handful of demonstrations in simulation. Learning from simulation drastically reduces the development cycle of behavior generation, and domain randomization techniques are leveraged to a…
▽ More
We present DemoStart, a novel auto-curriculum reinforcement learning method capable of learning complex manipulation behaviors on an arm equipped with a three-fingered robotic hand, from only a sparse reward and a handful of demonstrations in simulation. Learning from simulation drastically reduces the development cycle of behavior generation, and domain randomization techniques are leveraged to achieve successful zero-shot sim-to-real transfer. Transferred policies are learned directly from raw pixels from multiple cameras and robot proprioception. Our approach outperforms policies learned from demonstrations on the real robot and requires 100 times fewer demonstrations, collected in simulation. More details and videos in https://sites.google.com/view/demostart.
△ Less
Submitted 12 September, 2024; v1 submitted 10 September, 2024;
originally announced September 2024.
-
A Distributional View on Multi-Objective Policy Optimization
Authors:
Abbas Abdolmaleki,
Sandy H. Huang,
Leonard Hasenclever,
Michael Neunert,
H. Francis Song,
Martina Zambelli,
Murilo F. Martins,
Nicolas Heess,
Raia Hadsell,
Martin Riedmiller
Abstract:
Many real-world problems require trading off multiple competing objectives. However, these objectives are often in different units and/or scales, which can make it challenging for practitioners to express numerical preferences over objectives in their native units. In this paper we propose a novel algorithm for multi-objective reinforcement learning that enables setting desired preferences for obj…
▽ More
Many real-world problems require trading off multiple competing objectives. However, these objectives are often in different units and/or scales, which can make it challenging for practitioners to express numerical preferences over objectives in their native units. In this paper we propose a novel algorithm for multi-objective reinforcement learning that enables setting desired preferences for objectives in a scale-invariant way. We propose to learn an action distribution for each objective, and we use supervised learning to fit a parametric policy to a combination of these distributions. We demonstrate the effectiveness of our approach on challenging high-dimensional real and simulated robotics tasks, and show that setting different preferences in our framework allows us to trace out the space of nondominated solutions.
△ Less
Submitted 15 May, 2020;
originally announced May 2020.
-
Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning
Authors:
Sandy H. Huang,
Martina Zambelli,
Jackie Kay,
Murilo F. Martins,
Yuval Tassa,
Patrick M. Pilarski,
Raia Hadsell
Abstract:
Robots must know how to be gentle when they need to interact with fragile objects, or when the robot itself is prone to wear and tear. We propose an approach that enables deep reinforcement learning to train policies that are gentle, both during exploration and task execution. In a reward-based learning environment, a natural approach involves augmenting the (task) reward with a penalty for non-ge…
▽ More
Robots must know how to be gentle when they need to interact with fragile objects, or when the robot itself is prone to wear and tear. We propose an approach that enables deep reinforcement learning to train policies that are gentle, both during exploration and task execution. In a reward-based learning environment, a natural approach involves augmenting the (task) reward with a penalty for non-gentleness, which can be defined as excessive impact force. However, augmenting with only this penalty impairs learning: policies get stuck in a local optimum which avoids all contact with the environment. Prior research has shown that combining auxiliary tasks or intrinsic rewards can be beneficial for stabilizing and accelerating learning in sparse-reward domains, and indeed we find that introducing a surprise-based intrinsic reward does avoid the no-contact failure case. However, we show that a simple dynamics-based surprise is not as effective as penalty-based surprise. Penalty-based surprise, based on predicting forceful contacts, has a further benefit: it encourages exploration which is contact-rich yet gentle. We demonstrate the effectiveness of the approach using a complex, tendon-powered robot hand with tactile sensors. Videos are available at http://sites.google.com/view/gentlemanipulation.
△ Less
Submitted 20 March, 2019;
originally announced March 2019.
-
Simultaneously Learning Vision and Feature-based Control Policies for Real-world Ball-in-a-Cup
Authors:
Devin Schwab,
Tobias Springenberg,
Murilo F. Martins,
Thomas Lampe,
Michael Neunert,
Abbas Abdolmaleki,
Tim Hertweck,
Roland Hafner,
Francesco Nori,
Martin Riedmiller
Abstract:
We present a method for fast training of vision based control policies on real robots. The key idea behind our method is to perform multi-task Reinforcement Learning with auxiliary tasks that differ not only in the reward to be optimized but also in the state-space in which they operate. In particular, we allow auxiliary task policies to utilize task features that are available only at training-ti…
▽ More
We present a method for fast training of vision based control policies on real robots. The key idea behind our method is to perform multi-task Reinforcement Learning with auxiliary tasks that differ not only in the reward to be optimized but also in the state-space in which they operate. In particular, we allow auxiliary task policies to utilize task features that are available only at training-time. This allows for fast learning of auxiliary policies, which subsequently generate good data for training the main, vision-based control policies. This method can be seen as an extension of the Scheduled Auxiliary Control (SAC-X) framework. We demonstrate the efficacy of our method by using both a simulated and real-world Ball-in-a-Cup game controlled by a robot arm. In simulation, our approach leads to significant learning speed-ups when compared to standard SAC-X. On the real robot we show that the task can be learned from-scratch, i.e., with no transfer from simulation and no imitation learning. Videos of our learned policies running on the real robot can be found at https://sites.google.com/view/rss-2019-sawyer-bic/.
△ Less
Submitted 18 February, 2019; v1 submitted 12 February, 2019;
originally announced February 2019.
