-
SCOUT: A Situated and Multi-Modal Human-Robot Dialogue Corpus
Authors:
Stephanie M. Lukin,
Claire Bonial,
Matthew Marge,
Taylor Hudson,
Cory J. Hayes,
Kimberly A. Pollard,
Anthony Baker,
Ashley N. Foots,
Ron Artstein,
Felix Gervits,
Mitchell Abrams,
Cassidy Henry,
Lucia Donatelli,
Anton Leuski,
Susan G. Hill,
David Traum,
Clare R. Voss
Abstract:
We introduce the Situated Corpus Of Understanding Transactions (SCOUT), a multi-modal collection of human-robot dialogue in the task domain of collaborative exploration. The corpus was constructed from multiple Wizard-of-Oz experiments where human participants gave verbal instructions to a remotely-located robot to move and gather information about its surroundings. SCOUT contains 89,056 utterance…
▽ More
We introduce the Situated Corpus Of Understanding Transactions (SCOUT), a multi-modal collection of human-robot dialogue in the task domain of collaborative exploration. The corpus was constructed from multiple Wizard-of-Oz experiments where human participants gave verbal instructions to a remotely-located robot to move and gather information about its surroundings. SCOUT contains 89,056 utterances and 310,095 words from 278 dialogues averaging 320 utterances per dialogue. The dialogues are aligned with the multi-modal data streams available during the experiments: 5,785 images and 30 maps. The corpus has been annotated with Abstract Meaning Representation and Dialogue-AMR to identify the speaker's intent and meaning within an utterance, and with Transactional Units and Relations to track relationships between utterances to reveal patterns of the Dialogue Structure. We describe how the corpus and its annotations have been used to develop autonomous human-robot systems and enable research in open questions of how humans speak to robots. We release this corpus to accelerate progress in autonomous, situated, human-robot dialogue, especially in the context of navigation tasks where details about the environment need to be discovered.
△ Less
Submitted 19 November, 2024;
originally announced November 2024.
-
Human-Robot Dialogue Annotation for Multi-Modal Common Ground
Authors:
Claire Bonial,
Stephanie M. Lukin,
Mitchell Abrams,
Anthony Baker,
Lucia Donatelli,
Ashley Foots,
Cory J. Hayes,
Cassidy Henry,
Taylor Hudson,
Matthew Marge,
Kimberly A. Pollard,
Ron Artstein,
David Traum,
Clare R. Voss
Abstract:
In this paper, we describe the development of symbolic representations annotated on human-robot dialogue data to make dimensions of meaning accessible to autonomous systems participating in collaborative, natural language dialogue, and to enable common ground with human partners. A particular challenge for establishing common ground arises in remote dialogue (occurring in disaster relief or search…
▽ More
In this paper, we describe the development of symbolic representations annotated on human-robot dialogue data to make dimensions of meaning accessible to autonomous systems participating in collaborative, natural language dialogue, and to enable common ground with human partners. A particular challenge for establishing common ground arises in remote dialogue (occurring in disaster relief or search-and-rescue tasks), where a human and robot are engaged in a joint navigation and exploration task of an unfamiliar environment, but where the robot cannot immediately share high quality visual information due to limited communication constraints. Engaging in a dialogue provides an effective way to communicate, while on-demand or lower-quality visual information can be supplemented for establishing common ground. Within this paradigm, we capture propositional semantics and the illocutionary force of a single utterance within the dialogue through our Dialogue-AMR annotation, an augmentation of Abstract Meaning Representation. We then capture patterns in how different utterances within and across speaker floors relate to one another in our development of a multi-floor Dialogue Structure annotation schema. Finally, we begin to annotate and analyze the ways in which the visual modalities provide contextual information to the dialogue for overcoming disparities in the collaborators' understanding of the environment. We conclude by discussing the use-cases, architectures, and systems we have implemented from our annotations that enable physical robots to autonomously engage with humans in bi-directional dialogue and navigation.
△ Less
Submitted 19 November, 2024;
originally announced November 2024.
-
Balancing Efficiency and Coverage in Human-Robot Dialogue Collection
Authors:
Matthew Marge,
Claire Bonial,
Stephanie Lukin,
Cory Hayes,
Ashley Foots,
Ron Artstein,
Cassidy Henry,
Kimberly Pollard,
Carla Gordon,
Felix Gervits,
Anton Leuski,
Susan Hill,
Clare Voss,
David Traum
Abstract:
We describe a multi-phased Wizard-of-Oz approach to collecting human-robot dialogue in a collaborative search and navigation task. The data is being used to train an initial automated robot dialogue system to support collaborative exploration tasks. In the first phase, a wizard freely typed robot utterances to human participants. For the second phase, this data was used to design a GUI that includ…
▽ More
We describe a multi-phased Wizard-of-Oz approach to collecting human-robot dialogue in a collaborative search and navigation task. The data is being used to train an initial automated robot dialogue system to support collaborative exploration tasks. In the first phase, a wizard freely typed robot utterances to human participants. For the second phase, this data was used to design a GUI that includes buttons for the most common communications, and templates for communications with varying parameters. Comparison of the data gathered in these phases show that the GUI enabled a faster pace of dialogue while still maintaining high coverage of suitable responses, enabling more efficient targeted data collection, and improvements in natural language understanding using GUI-collected data. As a promising first step towards interactive learning, this work shows that our approach enables the collection of useful training data for navigation-based HRI tasks.
△ Less
Submitted 7 October, 2018; v1 submitted 3 October, 2018;
originally announced October 2018.
-
Laying Down the Yellow Brick Road: Development of a Wizard-of-Oz Interface for Collecting Human-Robot Dialogue
Authors:
Claire Bonial,
Matthew Marge,
Ron artstein,
Ashley Foots,
Felix Gervits,
Cory J. Hayes,
Cassidy Henry,
Susan G. Hill,
Anton Leuski,
Stephanie M. Lukin,
Pooja Moolchandani,
Kimberly A. Pollard,
David Traum,
Clare R. Voss
Abstract:
We describe the adaptation and refinement of a graphical user interface designed to facilitate a Wizard-of-Oz (WoZ) approach to collecting human-robot dialogue data. The data collected will be used to develop a dialogue system for robot navigation. Building on an interface previously used in the development of dialogue systems for virtual agents and video playback, we add templates with open param…
▽ More
We describe the adaptation and refinement of a graphical user interface designed to facilitate a Wizard-of-Oz (WoZ) approach to collecting human-robot dialogue data. The data collected will be used to develop a dialogue system for robot navigation. Building on an interface previously used in the development of dialogue systems for virtual agents and video playback, we add templates with open parameters which allow the wizard to quickly produce a wide variety of utterances. Our research demonstrates that this approach to data collection is viable as an intermediate step in developing a dialogue system for physical robots in remote locations from their users - a domain in which the human and robot need to regularly verify and update a shared understanding of the physical environment. We show that our WoZ interface and the fixed set of utterances and templates therein provide for a natural pace of dialogue with good coverage of the navigation domain.
△ Less
Submitted 17 October, 2017;
originally announced October 2017.
