-
Express Yourself: Enabling large-scale public events involving multi-human-swarm interaction for social applications with MOSAIX
Authors:
Merihan Alhafnawi,
Maca Gomez-Gutierrez,
Edmund R. Hunt,
Severin Lemaignan,
Paul O'Dowd,
Sabine Hauert
Abstract:
Robot swarms have the potential to help groups of people with social tasks, given their ability to scale to large numbers of robots and users. Developing multi-human-swarm interaction is therefore crucial to support multiple people interacting with the swarm simultaneously - which is an area that is scarcely researched, unlike single-human, single-robot or single-human, multi-robot interaction. Mo…
▽ More
Robot swarms have the potential to help groups of people with social tasks, given their ability to scale to large numbers of robots and users. Developing multi-human-swarm interaction is therefore crucial to support multiple people interacting with the swarm simultaneously - which is an area that is scarcely researched, unlike single-human, single-robot or single-human, multi-robot interaction. Moreover, most robots are still confined to laboratory settings. In this paper, we present our work with MOSAIX, a swarm of robot Tiles, that facilitated ideation at a science museum. 63 robots were used as a swarm of smart sticky notes, collecting input from the public and aggregating it based on themes, providing an evolving visualization tool that engaged visitors and fostered their participation. Our contribution lies in creating a large-scale (63 robots and 294 attendees) public event, with a completely decentralized swarm system in real-life settings. We also discuss learnings we obtained that might help future researchers create multi-human-swarm interaction with the public.
△ Less
Submitted 15 November, 2024;
originally announced November 2024.
-
Fusion in Context: A Multimodal Approach to Affective State Recognition
Authors:
Youssef Mohamed,
Severin Lemaignan,
Arzu Guneysu,
Patric Jensfelt,
Christian Smith
Abstract:
Accurate recognition of human emotions is a crucial challenge in affective computing and human-robot interaction (HRI). Emotional states play a vital role in shaping behaviors, decisions, and social interactions. However, emotional expressions can be influenced by contextual factors, leading to misinterpretations if context is not considered. Multimodal fusion, combining modalities like facial exp…
▽ More
Accurate recognition of human emotions is a crucial challenge in affective computing and human-robot interaction (HRI). Emotional states play a vital role in shaping behaviors, decisions, and social interactions. However, emotional expressions can be influenced by contextual factors, leading to misinterpretations if context is not considered. Multimodal fusion, combining modalities like facial expressions, speech, and physiological signals, has shown promise in improving affect recognition. This paper proposes a transformer-based multimodal fusion approach that leverages facial thermal data, facial action units, and textual context information for context-aware emotion recognition. We explore modality-specific encoders to learn tailored representations, which are then fused using additive fusion and processed by a shared transformer encoder to capture temporal dependencies and interactions. The proposed method is evaluated on a dataset collected from participants engaged in a tangible tabletop Pacman game designed to induce various affective states. Our results demonstrate the effectiveness of incorporating contextual information and multimodal fusion for affective state recognition.
△ Less
Submitted 18 September, 2024;
originally announced September 2024.
-
Socially Pertinent Robots in Gerontological Healthcare
Authors:
Xavier Alameda-Pineda,
Angus Addlesee,
Daniel Hernández García,
Chris Reinke,
Soraya Arias,
Federica Arrigoni,
Alex Auternaud,
Lauriane Blavette,
Cigdem Beyan,
Luis Gomez Camara,
Ohad Cohen,
Alessandro Conti,
Sébastien Dacunha,
Christian Dondrup,
Yoav Ellinson,
Francesco Ferro,
Sharon Gannot,
Florian Gras,
Nancie Gunson,
Radu Horaud,
Moreno D'Incà,
Imad Kimouche,
Séverin Lemaignan,
Oliver Lemon,
Cyril Liotard
, et al. (19 additional authors not shown)
Abstract:
Despite the many recent achievements in developing and deploying social robotics, there are still many underexplored environments and applications for which systematic evaluation of such systems by end-users is necessary. While several robotic platforms have been used in gerontological healthcare, the question of whether or not a social interactive robot with multi-modal conversational capabilitie…
▽ More
Despite the many recent achievements in developing and deploying social robotics, there are still many underexplored environments and applications for which systematic evaluation of such systems by end-users is necessary. While several robotic platforms have been used in gerontological healthcare, the question of whether or not a social interactive robot with multi-modal conversational capabilities will be useful and accepted in real-life facilities is yet to be answered. This paper is an attempt to partially answer this question, via two waves of experiments with patients and companions in a day-care gerontological facility in Paris with a full-sized humanoid robot endowed with social and conversational interaction capabilities. The software architecture, developed during the H2020 SPRING project, together with the experimental protocol, allowed us to evaluate the acceptability (AES) and usability (SUS) with more than 60 end-users. Overall, the users are receptive to this technology, especially when the robot perception and action skills are robust to environmental clutter and flexible to handle a plethora of different interactions.
△ Less
Submitted 11 February, 2025; v1 submitted 11 April, 2024;
originally announced April 2024.
-
End-User Development for Human-Robot Interaction
Authors:
Laura Stegner,
David Porfirio,
Laura M. Hiatt,
Séverin Lemaignan,
Ross Mead,
Bilge Mutlu
Abstract:
End-user development (EUD) represents a key step towards making robotics accessible for experts and nonexperts alike. Within academia, researchers investigate novel ways that EUD tools can capture, represent, visualize, analyze, and test developer intent. At the same time, industry researchers increasingly build and ship programming tools that enable customers to interact with their robots. Howeve…
▽ More
End-user development (EUD) represents a key step towards making robotics accessible for experts and nonexperts alike. Within academia, researchers investigate novel ways that EUD tools can capture, represent, visualize, analyze, and test developer intent. At the same time, industry researchers increasingly build and ship programming tools that enable customers to interact with their robots. However, despite this growing interest, the role of EUD within HRI is not well defined. EUD struggles to situate itself within a growing array of alternative approaches to application development, such as robot learning and teleoperation. EUD further struggles due to the wide range of individuals who can be considered end users, such as independent third-party application developers, consumers, hobbyists, or even employees of the robot manufacturer. Key questions remain such as how EUD is justified over alternate approaches to application development, which contexts EUD is most suited for, who the target users of an EUD system are, and where interaction between a human and a robot takes place, amongst many other questions. We seek to address these challenges and questions by organizing the first End-User Development for Human-Robot Interaction (EUD4HRI) workshop at the 2024 International Conference of Human-Robot Interaction. The workshop will bring together researchers with a wide range of expertise across academia and industry, spanning perspectives from multiple subfields of robotics, with the primary goal being a consensus of perspectives about the role that EUD must play within human-robot interaction.
△ Less
Submitted 27 February, 2024;
originally announced February 2024.
-
Bio-Inspired Grasping Controller for Sensorized 2-DoF Grippers
Authors:
Luca Lach,
Séverin Lemaignan,
Francesco Ferro,
Helge Ritter,
Robert Haschke
Abstract:
We present a holistic grasping controller, combining free-space position control and in-contact force-control for reliable grasping given uncertain object pose estimates. Employing tactile fingertip sensors, undesired object displacement during grasping is minimized by pausing the finger closing motion for individual joints on first contact until force-closure is established. While holding an obje…
▽ More
We present a holistic grasping controller, combining free-space position control and in-contact force-control for reliable grasping given uncertain object pose estimates. Employing tactile fingertip sensors, undesired object displacement during grasping is minimized by pausing the finger closing motion for individual joints on first contact until force-closure is established. While holding an object, the controller is compliant with external forces to avoid high internal object forces and prevent object damage. Gravity as an external force is explicitly considered and compensated for, thus preventing gravity-induced object drift. We evaluate the controller in two experiments on the TIAGo robot and its parallel-jaw gripper proving the effectiveness of the approach for robust grasping and minimizing object displacement. In a series of ablation studies, we demonstrate the utility of the individual controller components.
△ Less
Submitted 13 November, 2023;
originally announced November 2023.
-
Placing by Touching: An empirical study on the importance of tactile sensing for precise object placing
Authors:
Luca Lach,
Niklas Funk,
Robert Haschke,
Severin Lemaignan,
Helge Joachim Ritter,
Jan Peters,
Georgia Chalvatzaki
Abstract:
This work deals with a practical everyday problem: stable object placement on flat surfaces starting from unknown initial poses. Common object-placing approaches require either complete scene specifications or extrinsic sensor measurements, e.g., cameras, that occasionally suffer from occlusions. We propose a novel approach for stable object placing that combines tactile feedback and proprioceptiv…
▽ More
This work deals with a practical everyday problem: stable object placement on flat surfaces starting from unknown initial poses. Common object-placing approaches require either complete scene specifications or extrinsic sensor measurements, e.g., cameras, that occasionally suffer from occlusions. We propose a novel approach for stable object placing that combines tactile feedback and proprioceptive sensing. We devise a neural architecture that estimates a rotation matrix, resulting in a corrective gripper movement that aligns the object with the placing surface for the subsequent object manipulation. We compare models with different sensing modalities, such as force-torque and an external motion capture system, in real-world object placing tasks with different objects. The experimental evaluation of our placing policies with a set of unseen everyday objects reveals significant generalization of our proposed pipeline, suggesting that tactile sensing plays a vital role in the intrinsic understanding of robotic dexterous object manipulation. Code, models, and supplementary videos are available at https://sites.google.com/view/placing-by-touching.
△ Less
Submitted 27 November, 2023; v1 submitted 5 October, 2022;
originally announced October 2022.
-
UNICEF Guidance on AI for Children: Application to the Design of a Social Robot For and With Autistic Children
Authors:
Séverin Lemaignan,
Nigel Newbutt,
Louis Rice,
Joe Daly,
Vicky Charisi
Abstract:
For a period of three weeks in June 2021, we embedded a social robot (Softbank Pepper) in a Special Educational Needs (SEN) school, with a focus on supporting the well-being of autistic children. Our methodology to design and embed the robot among this vulnerable population follows a comprehensive participatory approach. We used the research project as a test-bed to demonstrate in a complex real-w…
▽ More
For a period of three weeks in June 2021, we embedded a social robot (Softbank Pepper) in a Special Educational Needs (SEN) school, with a focus on supporting the well-being of autistic children. Our methodology to design and embed the robot among this vulnerable population follows a comprehensive participatory approach. We used the research project as a test-bed to demonstrate in a complex real-world environment the importance and suitability of the nine UNICEF guidelines on AI for Children. The UNICEF guidelines on AI for Children closely align with several of the UN goals for sustainable development, and, as such, we report here our contribution to these goals.
△ Less
Submitted 27 August, 2021;
originally announced August 2021.
-
LEADOR: A Method for End-to-End Participatory Design of Autonomous Social Robots
Authors:
Katie Winkle,
Emmanuel Senft,
Séverin Lemaignan
Abstract:
Participatory Design (PD) in Human-Robot Interaction (HRI) typically remains limited to the early phases of development, with subsequent robot behaviours then being hardcoded by engineers or utilised in Wizard-of-Oz (WoZ) systems that rarely achieve autonomy. We present LEADOR (Led-by-Experts Automation and Design Of Robots) an end-to-end PD methodology for domain expert co-design, automation and…
▽ More
Participatory Design (PD) in Human-Robot Interaction (HRI) typically remains limited to the early phases of development, with subsequent robot behaviours then being hardcoded by engineers or utilised in Wizard-of-Oz (WoZ) systems that rarely achieve autonomy. We present LEADOR (Led-by-Experts Automation and Design Of Robots) an end-to-end PD methodology for domain expert co-design, automation and evaluation of social robots. LEADOR starts with typical PD to co-design the interaction specifications and state and action space of the robot. It then replaces traditional offline programming or WoZ by an in-situ, online teaching phase where the domain expert can live-program or teach the robot how to behave while being embedded in the interaction context. We believe that this live teaching can be best achieved by adding a learning component to a WoZ setup, to capture experts' implicit knowledge, as they intuitively respond to the dynamics of the situation. The robot progressively learns an appropriate, expert-approved policy, ultimately leading to full autonomy, even in sensitive and/or ill-defined environments. However, LEADOR is agnostic to the exact technical approach used to facilitate this learning process. The extensive inclusion of the domain expert(s) in robot design represents established responsible innovation practice, lending credibility to the system both during the teaching phase and when operating autonomously. The combination of this expert inclusion with the focus on in-situ development also means LEADOR supports a mutual shaping approach to social robotics. We draw on two previously published, foundational works from which this (generalisable) methodology has been derived in order to demonstrate the feasibility and worth of this approach, provide concrete examples in its application and identify limitations and opportunities when applying this framework in new environments.
△ Less
Submitted 14 May, 2021; v1 submitted 5 May, 2021;
originally announced May 2021.
-
On Determinism of Game Engines used for Simulation-based Autonomous Vehicle Verification
Authors:
Greg Chance,
Abanoub Ghobrial,
Kevin McAreavey,
Severin Lemaignan,
Tony Pipe,
Kerstin Eder
Abstract:
Game engines are increasingly used as simulation platforms by the autonomous vehicle (AV) community to develop vehicle control systems and test environments. A key requirement for simulation-based development and verification is determinism, since a deterministic process will always produce the same output given the same initial conditions and event history. Thus, in a deterministic simulation env…
▽ More
Game engines are increasingly used as simulation platforms by the autonomous vehicle (AV) community to develop vehicle control systems and test environments. A key requirement for simulation-based development and verification is determinism, since a deterministic process will always produce the same output given the same initial conditions and event history. Thus, in a deterministic simulation environment, tests are rendered repeatable and yield simulation results that are trustworthy and straightforward to debug. However, game engines are seldom deterministic. This paper reviews and identifies the potential causes of non-deterministic behaviours in game engines. A case study using CARLA, an open-source autonomous driving simulation environment powered by Unreal Engine, is presented to highlight its inherent shortcomings in providing sufficient precision in experimental results. Different configurations and utilisations of the software and hardware are explored to determine an operational domain where the simulation precision is sufficiently low i.e.\ variance between repeated executions becomes negligible for development and testing work. Finally, a method of a general nature is proposed, that can be used to find the domains of permissible variance in game engine simulations for any given system configuration.
△ Less
Submitted 9 February, 2022; v1 submitted 7 April, 2021;
originally announced April 2021.
-
ROS for Human-Robot Interaction
Authors:
Youssef Mohamed,
Séverin Lemaignan
Abstract:
Integrating real-time, complex social signal processing into robotic systems -- especially in real-world, multi-party interaction situations -- is a challenge faced by many in the Human-Robot Interaction (HRI) community. The difficulty is compounded by the lack of any standard model for human representation that would facilitate the development and interoperability of social perception components…
▽ More
Integrating real-time, complex social signal processing into robotic systems -- especially in real-world, multi-party interaction situations -- is a challenge faced by many in the Human-Robot Interaction (HRI) community. The difficulty is compounded by the lack of any standard model for human representation that would facilitate the development and interoperability of social perception components and pipelines. We introduce in this paper a set of conventions and standard interfaces for HRI scenarios, designed to be used with the Robot Operating System (ROS). It directly aims at promoting interoperability and re-usability of core functionality between the many HRI-related software tools, from skeleton tracking, to face recognition, to natural language processing. Importantly, these interfaces are designed to be relevant to a broad range of HRI applications, from high-level crowd simulation, to group-level social interaction modelling, to detailed modelling of human kinematics. We demonstrate these interface by providing a reference pipeline implementation, packaged to be easily downloaded and evaluated by the community.
△ Less
Submitted 27 December, 2020;
originally announced December 2020.
-
An Agency-Directed Approach to Test Generation for Simulation-based Autonomous Vehicle Verification
Authors:
Greg Chance,
Abanoub Ghobrial,
Severin Lemaignan,
Tony Pipe,
Kerstin Eder
Abstract:
Simulation-based verification is beneficial for assessing otherwise dangerous or costly on-road testing of autonomous vehicles (AV). This paper addresses the challenge of efficiently generating effective tests for simulation-based AV verification using software testing agents. The multi-agent system (MAS) programming paradigm offers rational agency, causality and strategic planning between multipl…
▽ More
Simulation-based verification is beneficial for assessing otherwise dangerous or costly on-road testing of autonomous vehicles (AV). This paper addresses the challenge of efficiently generating effective tests for simulation-based AV verification using software testing agents. The multi-agent system (MAS) programming paradigm offers rational agency, causality and strategic planning between multiple agents. We exploit these aspects for test generation, focusing in particular on the generation of tests that trigger the precondition of an assertion. On the example of a key assertion we show that, by encoding a variety of different behaviours respondent to the agent's perceptions of the test environment, the agency-directed approach generates twice as many effective tests than pseudo-random test generation, while being both efficient and robust. Moreover, agents can be encoded to behave naturally without compromising the effectiveness of test generation. Our results suggest that generating tests using agency-directed testing significantly improves upon random and simultaneously provides more realistic driving scenarios.
△ Less
Submitted 14 February, 2020; v1 submitted 11 December, 2019;
originally announced December 2019.
-
The Free-play Sandbox: a Methodology for the Evaluation of Social Robotics and a Dataset of Social Interactions
Authors:
Séverin Lemaignan,
Charlotte Edmunds,
Emmanuel Senft,
Tony Belpaeme
Abstract:
Evaluating human-robot social interactions in a rigorous manner is notoriously difficult: studies are either conducted in labs with constrained protocols to allow for robust measurements and a degree of replicability, but at the cost of ecological validity; or in the wild, which leads to superior experimental realism, but often with limited replicability and at the expense of rigorous interaction…
▽ More
Evaluating human-robot social interactions in a rigorous manner is notoriously difficult: studies are either conducted in labs with constrained protocols to allow for robust measurements and a degree of replicability, but at the cost of ecological validity; or in the wild, which leads to superior experimental realism, but often with limited replicability and at the expense of rigorous interaction metrics.
We introduce a novel interaction paradigm, designed to elicit rich and varied social interactions while having desirable scientific properties (replicability, clear metrics, possibility of either autonomous or Wizard-of-Oz robot behaviours). This paradigm focuses on child-robot interactions, and builds on a sandboxed free-play environment. We present the rationale and design of the interaction paradigm, its methodological and technical aspects (including the open-source implementation of the software platform), as well as two large open datasets acquired with this paradigm, and meant to act as experimental baselines for future research.
△ Less
Submitted 6 December, 2017;
originally announced December 2017.
-
2nd Workshop on Cognitive Architectures for Social Human-Robot Interaction 2016 (CogArch4sHRI 2016)
Authors:
Paul Baxter,
J. Gregory Trafton,
Severin Lemaignan
Abstract:
This volume is the proceedings of the 2nd workshop on Cognitive Architectures for Social Human-Robot Interaction, held at the ACM/IEEE HRI 2016 conference, which took place on Monday 7th March 2016, in Christchurch, New Zealand.
Organised by Paul Baxter (Plymouth University, U.K.), J. Gregory Trafton (Naval Research Laboratory, USA), and Severin Lemaignan (Plymouth University, U.K.).
This volume is the proceedings of the 2nd workshop on Cognitive Architectures for Social Human-Robot Interaction, held at the ACM/IEEE HRI 2016 conference, which took place on Monday 7th March 2016, in Christchurch, New Zealand.
Organised by Paul Baxter (Plymouth University, U.K.), J. Gregory Trafton (Naval Research Laboratory, USA), and Severin Lemaignan (Plymouth University, U.K.).
△ Less
Submitted 4 February, 2016;
originally announced February 2016.
