The Bystander Affect Detection (BAD) Dataset for Failure Detection in HRI
Authors:
Alexandra Bremers,
Maria Teresa Parreira,
Xuanyu Fang,
Natalie Friedman,
Adolfo Ramirez-Aristizabal,
Alexandria Pabst,
Mirjana Spasojevic,
Michael Kuniavsky,
Wendy Ju
Abstract:
For a robot to repair its own error, it must first know it has made a mistake. One way that people detect errors is from the implicit reactions from bystanders -- their confusion, smirks, or giggles clue us in that something unexpected occurred. To enable robots to detect and act on bystander responses to task failures, we developed a novel method to elicit bystander responses to human and robot e…
▽ More
For a robot to repair its own error, it must first know it has made a mistake. One way that people detect errors is from the implicit reactions from bystanders -- their confusion, smirks, or giggles clue us in that something unexpected occurred. To enable robots to detect and act on bystander responses to task failures, we developed a novel method to elicit bystander responses to human and robot errors. Using 46 different stimulus videos featuring a variety of human and machine task failures, we collected a total of 2452 webcam videos of human reactions from 54 participants. To test the viability of the collected data, we used the bystander reaction dataset as input to a deep-learning model, BADNet, to predict failure occurrence. We tested different data labeling methods and learned how they affect model performance, achieving precisions above 90%. We discuss strategies to model bystander reactions and predict failure and how this approach can be used in real-world robotic deployments to detect errors and improve robot performance. As part of this work, we also contribute with the "Bystander Affect Detection" (BAD) dataset of bystander reactions, supporting the development of better prediction models.
△ Less
Submitted 8 March, 2023;
originally announced March 2023.
Using Social Cues to Recognize Task Failures for HRI: Overview, State-of-the-Art, and Future Directions
Authors:
Alexandra Bremers,
Alexandria Pabst,
Maria Teresa Parreira,
Wendy Ju
Abstract:
Robots that carry out tasks and interact in complex environments will inevitably commit errors. Error detection is thus an essential ability for robots to master to work efficiently and productively. People can leverage social feedback to get an indication of whether an action was successful or not. With advances in computing and artificial intelligence (AI), it is increasingly possible for robots…
▽ More
Robots that carry out tasks and interact in complex environments will inevitably commit errors. Error detection is thus an essential ability for robots to master to work efficiently and productively. People can leverage social feedback to get an indication of whether an action was successful or not. With advances in computing and artificial intelligence (AI), it is increasingly possible for robots to achieve a similar capability of collecting social feedback. In this work, we take this one step further and propose a framework for how social cues can be used as feedback signals to recognize task failures for human-robot interaction (HRI). Our proposed framework sets out a research agenda based on insights from the literature on behavioral science, human-robot interaction, and machine learning to focus on three areas: 1) social cues as feedback (from behavioral science), 2) recognizing task failures in robots (from HRI), and 3) approaches for autonomous detection of HRI task failures based on social cues (from machine learning). We propose a taxonomy of error detection based on self-awareness and social feedback. Finally, we provide recommendations for HRI researchers and practitioners interested in developing robots that detect task errors using human social cues. This article is intended for interdisciplinary HRI researchers and practitioners, where the third theme of our analysis provides more technical details aiming toward the practical implementation of these systems.
△ Less
Submitted 29 May, 2024; v1 submitted 27 January, 2023;
originally announced January 2023.
