-
Empathetic Conversational Agents: Utilizing Neural and Physiological Signals for Enhanced Empathetic Interactions
Authors:
Nastaran Saffaryazdi,
Tamil Selvan Gunasekaran,
Kate Laveys,
Elizabeth Broadbent,
Mark Billinghurst
Abstract:
Conversational agents (CAs) are revolutionizing human-computer interaction by evolving from text-based chatbots to empathetic digital humans (DHs) capable of rich emotional expressions. This paper explores the integration of neural and physiological signals into the perception module of CAs to enhance empathetic interactions. By leveraging these cues, the study aims to detect emotions in real-time…
▽ More
Conversational agents (CAs) are revolutionizing human-computer interaction by evolving from text-based chatbots to empathetic digital humans (DHs) capable of rich emotional expressions. This paper explores the integration of neural and physiological signals into the perception module of CAs to enhance empathetic interactions. By leveraging these cues, the study aims to detect emotions in real-time and generate empathetic responses and expressions. We conducted a user study where participants engaged in conversations with a DH about emotional topics. The DH responded and displayed expressions by mirroring detected emotions in real-time using neural and physiological cues. The results indicate that participants experienced stronger emotions and greater engagement during interactions with the Empathetic DH, demonstrating the effectiveness of incorporating neural and physiological signals for real-time emotion recognition. However, several challenges were identified, including recognition accuracy, emotional transition speeds, individual personality effects, and limitations in voice tone modulation. Addressing these challenges is crucial for further refining Empathetic DHs and fostering meaningful connections between humans and artificial entities. Overall, this research advances human-agent interaction and highlights the potential of real-time neural and physiological emotion recognition in creating empathetic DHs.
△ Less
Submitted 14 January, 2025;
originally announced January 2025.
-
Using Virtual Reality as a Simulation Tool for Augmented Reality Virtual Windows: Effects on Cognitive Workload and Task Performance
Authors:
Tianyu Liu,
Weiping He,
Mark Billinghurst
Abstract:
Virtual content in Augmented Reality (AR) applications can be constructed according to the designer's requirements, but real environments, are difficult to be accurate control or completely reproduce. This makes it difficult to prototype AR applications for certain real environments. One way to address this issue is to use Virtual Reality (VR) to simulate an AR system, enabling the design of contr…
▽ More
Virtual content in Augmented Reality (AR) applications can be constructed according to the designer's requirements, but real environments, are difficult to be accurate control or completely reproduce. This makes it difficult to prototype AR applications for certain real environments. One way to address this issue is to use Virtual Reality (VR) to simulate an AR system, enabling the design of controlled experiments and conducting usability evaluations. However, the effectiveness of using VR to simulate AR has not been well studied. In this paper, we report on a user study (N=20) conducted to investigate the impact of using an VR simulation of AR on participants' task performance and cognitive workload (CWL). Participants performed several office tasks in an AR scene with virtual monitors and then again in the VR-simulated AR scene. While using the interfaces CWL was measured with Electroencephalography (EEG) data and a subjective questionnaire. Results showed that frequent visual checks on the keyboard resulted in decreased task performance and increased cognitive workload. This study found that using AR centered on virtual monitor can be effectively simulated using VR. However, there is more research that can be done, so we also report on the study limitations and directions for future work.
△ Less
Submitted 24 September, 2024;
originally announced September 2024.
-
Explorations in Designing Virtual Environments for Remote Counselling
Authors:
Jiashuo Cao,
Wujie Gao,
Yun Suen Pai,
Simon Hoermann,
Chen Li,
Nilufar Baghaei,
Mark Billinghurst
Abstract:
The advent of technology-enhanced interventions has significantly transformed mental health services, offering new opportunities for delivering psychotherapy, particularly in remote settings. This paper reports on a pilot study exploring the use of Virtual Reality (VR) as a medium for remote counselling. The study involved four experienced psychotherapists who evaluated three different virtual env…
▽ More
The advent of technology-enhanced interventions has significantly transformed mental health services, offering new opportunities for delivering psychotherapy, particularly in remote settings. This paper reports on a pilot study exploring the use of Virtual Reality (VR) as a medium for remote counselling. The study involved four experienced psychotherapists who evaluated three different virtual environments designed to support remote counselling. Through thematic analysis of interviews and feedback, we identified key factors that could be critical for designing effective virtual environments for counselling. These include the creation of clear boundaries, customization to meet specific therapeutic needs, and the importance of aligning the environment with various therapeutic approaches. Our findings suggest that VR can enhance the sense of presence and engagement in remote therapy, potentially improving the therapeutic relationship. In the paper we also outline areas for future research based on these pilot study results.
△ Less
Submitted 12 September, 2024;
originally announced September 2024.
-
Large Language Models for Automatic Detection of Sensitive Topics
Authors:
Ruoyu Wen,
Stephanie Elena Crowe,
Kunal Gupta,
Xinyue Li,
Mark Billinghurst,
Simon Hoermann,
Dwain Allan,
Alaeddin Nassani,
Thammathip Piumsomboon
Abstract:
Sensitive information detection is crucial in content moderation to maintain safe online communities. Assisting in this traditionally manual process could relieve human moderators from overwhelming and tedious tasks, allowing them to focus solely on flagged content that may pose potential risks. Rapidly advancing large language models (LLMs) are known for their capability to understand and process…
▽ More
Sensitive information detection is crucial in content moderation to maintain safe online communities. Assisting in this traditionally manual process could relieve human moderators from overwhelming and tedious tasks, allowing them to focus solely on flagged content that may pose potential risks. Rapidly advancing large language models (LLMs) are known for their capability to understand and process natural language and so present a potential solution to support this process. This study explores the capabilities of five LLMs for detecting sensitive messages in the mental well-being domain within two online datasets and assesses their performance in terms of accuracy, precision, recall, F1 scores, and consistency. Our findings indicate that LLMs have the potential to be integrated into the moderation workflow as a convenient and precise detection tool. The best-performing model, GPT-4o, achieved an average accuracy of 99.5\% and an F1-score of 0.99. We discuss the advantages and potential challenges of using LLMs in the moderation workflow and suggest that future research should address the ethical considerations of utilising this technology.
△ Less
Submitted 2 September, 2024;
originally announced September 2024.
-
Integrating Physiological Data with Large Language Models for Empathic Human-AI Interaction
Authors:
Poorvesh Dongre,
Majid Behravan,
Kunal Gupta,
Mark Billinghurst,
Denis Gračanin
Abstract:
This paper explores enhancing empathy in Large Language Models (LLMs) by integrating them with physiological data. We propose a physiological computing approach that includes developing deep learning models that use physiological data for recognizing psychological states and integrating the predicted states with LLMs for empathic interaction. We showcase the application of this approach in an Empa…
▽ More
This paper explores enhancing empathy in Large Language Models (LLMs) by integrating them with physiological data. We propose a physiological computing approach that includes developing deep learning models that use physiological data for recognizing psychological states and integrating the predicted states with LLMs for empathic interaction. We showcase the application of this approach in an Empathic LLM (EmLLM) chatbot for stress monitoring and control. We also discuss the results of a pilot study that evaluates this EmLLM chatbot based on its ability to accurately predict user stress, provide human-like responses, and assess the therapeutic alliance with the user.
△ Less
Submitted 14 April, 2024;
originally announced April 2024.
-
Putting Our Minds Together: Iterative Exploration for Collaborative Mind Mapping
Authors:
Ying Yang,
Tim Dwyer,
Zachari Swiecki,
Benjamin Lee,
Michael Wybrow,
Maxime Cordeil,
Teresa Wulandari,
Bruce H. Thomas,
Mark Billinghurst
Abstract:
We delineate the development of a mind-mapping system designed concurrently for both VR and desktop platforms. Employing an iterative methodology with groups of users, we systematically examined and improved various facets of our system, including interactions, communication mechanisms and gamification elements, to streamline the mind-mapping process while augmenting situational awareness and prom…
▽ More
We delineate the development of a mind-mapping system designed concurrently for both VR and desktop platforms. Employing an iterative methodology with groups of users, we systematically examined and improved various facets of our system, including interactions, communication mechanisms and gamification elements, to streamline the mind-mapping process while augmenting situational awareness and promoting active engagement among collaborators. We also report our observational findings on these facets from this iterative design process.
△ Less
Submitted 23 March, 2024; v1 submitted 20 March, 2024;
originally announced March 2024.
-
VR.net: A Real-world Dataset for Virtual Reality Motion Sickness Research
Authors:
Elliott Wen,
Chitralekha Gupta,
Prasanth Sasikumar,
Mark Billinghurst,
James Wilmott,
Emily Skow,
Arindam Dey,
Suranga Nanayakkara
Abstract:
Researchers have used machine learning approaches to identify motion sickness in VR experience. These approaches demand an accurately-labeled, real-world, and diverse dataset for high accuracy and generalizability. As a starting point to address this need, we introduce `VR.net', a dataset offering approximately 12-hour gameplay videos from ten real-world games in 10 diverse genres. For each video…
▽ More
Researchers have used machine learning approaches to identify motion sickness in VR experience. These approaches demand an accurately-labeled, real-world, and diverse dataset for high accuracy and generalizability. As a starting point to address this need, we introduce `VR.net', a dataset offering approximately 12-hour gameplay videos from ten real-world games in 10 diverse genres. For each video frame, a rich set of motion sickness-related labels, such as camera/object movement, depth field, and motion flow, are accurately assigned. Building such a dataset is challenging since manual labeling would require an infeasible amount of time. Instead, we utilize a tool to automatically and precisely extract ground truth data from 3D engines' rendering pipelines without accessing VR games' source code. We illustrate the utility of VR.net through several applications, such as risk factor detection and sickness level prediction. We continuously expand VR.net and envision its next version offering 10X more data than the current form. We believe that the scale, accuracy, and diversity of VR.net can offer unparalleled opportunities for VR motion sickness research and beyond.
△ Less
Submitted 5 June, 2023;
originally announced June 2023.
-
Synthesizing Affective Neurophysiological Signals Using Generative Models: A Review Paper
Authors:
Alireza F. Nia,
Vanessa Tang,
Gonzalo Maso Talou,
Mark Billinghurst
Abstract:
The integration of emotional intelligence in machines is an important step in advancing human-computer interaction. This demands the development of reliable end-to-end emotion recognition systems. However, the scarcity of public affective datasets presents a challenge. In this literature review, we emphasize the use of generative models to address this issue in neurophysiological signals, particul…
▽ More
The integration of emotional intelligence in machines is an important step in advancing human-computer interaction. This demands the development of reliable end-to-end emotion recognition systems. However, the scarcity of public affective datasets presents a challenge. In this literature review, we emphasize the use of generative models to address this issue in neurophysiological signals, particularly Electroencephalogram (EEG) and Functional Near-Infrared Spectroscopy (fNIRS). We provide a comprehensive analysis of different generative models used in the field, examining their input formulation, deployment strategies, and methodologies for evaluating the quality of synthesized data. This review serves as a comprehensive overview, offering insights into the advantages, challenges, and promising future directions in the application of generative models in emotion recognition systems. Through this review, we aim to facilitate the progression of neurophysiological data augmentation, thereby supporting the development of more efficient and reliable emotion recognition systems.
△ Less
Submitted 5 June, 2023;
originally announced June 2023.
-
Wish You Were Here: Mental and Physiological Effects of Remote Music Collaboration in Mixed Reality
Authors:
Ruben Schlagowski,
Dariia Nazarenko,
Yekta Can,
Kunal Gupta,
Silvan Mertes,
Mark Billinghurst,
Elisabeth André
Abstract:
With face-to-face music collaboration being severely limited during the recent pandemic, mixed reality technologies and their potential to provide musicians a feeling of "being there" with their musical partner can offer tremendous opportunities. In order to assess this potential, we conducted a laboratory study in which musicians made music together in real-time while simultaneously seeing their…
▽ More
With face-to-face music collaboration being severely limited during the recent pandemic, mixed reality technologies and their potential to provide musicians a feeling of "being there" with their musical partner can offer tremendous opportunities. In order to assess this potential, we conducted a laboratory study in which musicians made music together in real-time while simultaneously seeing their jamming partner's mixed reality point cloud via a head-mounted display and compared mental effects such as flow, affect, and co-presence to an audio-only baseline. In addition, we tracked the musicians' physiological signals and evaluated their features during times of self-reported flow. For users jamming in mixed reality, we observed a significant increase in co-presence. Regardless of the condition (mixed reality or audio-only), we observed an increase in positive affect after jamming remotely. Furthermore, we identified heart rate and HF/LF as promising features for classifying the flow state musicians experienced while making music together.
△ Less
Submitted 23 January, 2023;
originally announced January 2023.
-
Towards Immersive Collaborative Sensemaking
Authors:
Ying Yang,
Tim Dwyer,
Michael Wybrow,
Benjamin Lee,
Maxime Cordeil,
Mark Billinghurst,
Bruce H. Thomas
Abstract:
When collaborating face-to-face, people commonly use the surfaces and spaces around them to perform sensemaking tasks, such as spatially organising documents, notes or images. However, when people collaborate remotely using desktop interfaces they no longer feel like they are sharing the same space. This limitation may be overcome through collaboration in immersive environments, which simulate the…
▽ More
When collaborating face-to-face, people commonly use the surfaces and spaces around them to perform sensemaking tasks, such as spatially organising documents, notes or images. However, when people collaborate remotely using desktop interfaces they no longer feel like they are sharing the same space. This limitation may be overcome through collaboration in immersive environments, which simulate the physical in-person experience. In this paper, we report on a between-groups study comparing collaborations on image organisation tasks, in an immersive Virtual Reality (VR) environment to more conventional desktop conferencing. Collecting data from 40 subjects in groups of four, we measured task performance, user behaviours, collaboration engagement and awareness. Overall, the VR and desktop interface resulted in similar speed, accuracy and social presence rating, but we observed more conversations and interaction with objects, and more equal contributions to the interaction from participants within groups in VR. We also identified differences in coordination and collaborative awareness behaviours between VR and desktop platforms. We report on a set of systematic measures for assessing VR collaborative experience and a new analysis tool that we have developed to capture user behaviours in collaborative setting. Finally, we provide design considerations and directions for future work.
△ Less
Submitted 14 October, 2022;
originally announced October 2022.
-
Prototyping a Virtual Agent for Pre-school English Teaching
Authors:
Eduardo Benitez Sandoval,
Diego Vazquez Rojas,
Clarissa A. Parada Cereceres,
Alvaro Anzueto Rios,
Amit Barde,
Mark Billinghurst
Abstract:
This paper describes a case study and the insights gained from prototyping an Intelligent Virtual Agent (IVA) for English vocabulary building for Spanish-speaking preschool children. After an initial exploration to evaluate the feasibility of developing an IVA, we followed a Human-Centered Design (HCD) approach to create a prototype. We report on the multidisciplinary process used that incorporate…
▽ More
This paper describes a case study and the insights gained from prototyping an Intelligent Virtual Agent (IVA) for English vocabulary building for Spanish-speaking preschool children. After an initial exploration to evaluate the feasibility of developing an IVA, we followed a Human-Centered Design (HCD) approach to create a prototype. We report on the multidisciplinary process used that incorporated two well-known educative concepts: gamification and story-telling as the main components for engagement. Our results suggest that a multidisciplinary approach to developing an educational IVA is effective. We report on the relevant aspects of the ideation and design processes that informed the vision and mission of the project.
△ Less
Submitted 8 February, 2022;
originally announced February 2022.
-
Reducing beam hardening effects and metal artefacts using Medipix3RX: With applications from biomaterial science
Authors:
K. Rajendran,
M. F. Walsh,
N. J. A. de Ruiter,
A. I. Chernoglazov,
R. K. Panta,
A. P. H. Butler,
P. H. Butler,
S. T. Bell,
N. G. Anderson,
T. B. F. Woodfield,
S. J. Tredinnick,
J. L. Healy,
C. J. Bateman,
R. Aamir,
R. M. N. Doesburg,
P. F. Renaud,
S. P. Gieseg,
D. J. Smithies,
J. L. Mohr,
V. B. H. Mandalika,
A. M. T. Opie,
N. J. Cook,
J. P. Ronaldson,
S. J. Nik,
A. Atharifard
, et al. (6 additional authors not shown)
Abstract:
This paper discusses methods for reducing beam hardening effects using spectral data for biomaterial applications. A small-animal spectral scanner operating in the diagnostic energy range was used. We investigate the use of photon-processing features of the Medipix3RX ASIC in reducing beam hardening and associated artefacts. A fully operational charge summing mode was used during the imaging routi…
▽ More
This paper discusses methods for reducing beam hardening effects using spectral data for biomaterial applications. A small-animal spectral scanner operating in the diagnostic energy range was used. We investigate the use of photon-processing features of the Medipix3RX ASIC in reducing beam hardening and associated artefacts. A fully operational charge summing mode was used during the imaging routine. We present spectral data collected for metal alloy samples, its analysis using algebraic 3D reconstruction software and volume visualisation using a custom volume rendering software. Narrow high energy acquisition using the photon-processing detector revealed substantial reduction in beam hardening effects and metal artefacts.
△ Less
Submitted 20 November, 2013;
originally announced November 2013.
-
MARS spectral molecular imaging of lamb tissue: data collection and image analysis
Authors:
R Aamir,
A Chernoglazov,
C J Bateman,
A P H Butler,
P H Butler,
N G Anderson,
S T Bell,
R K Panta,
J L Healy,
J L Mohr,
K Rajendran,
M F Walsh,
N de Ruiter,
S P Gieseg,
T Woodfield,
P F Renaud,
L Brooke,
S Abdul-Majid,
M Clyne,
R Glendenning,
P J Bones,
M Billinghurst,
C Bartneck,
H Mandalika,
R Grasset
, et al. (13 additional authors not shown)
Abstract:
Spectral molecular imaging is a new imaging technique able to discriminate and quantify different components of tissue simultaneously at high spatial and high energy resolution. Our MARS scanner is an x-ray based small animal CT system designed to be used in the diagnostic energy range (20 to 140 keV). In this paper, we demonstrate the use of the MARS scanner, equipped with the Medipix3RX spectros…
▽ More
Spectral molecular imaging is a new imaging technique able to discriminate and quantify different components of tissue simultaneously at high spatial and high energy resolution. Our MARS scanner is an x-ray based small animal CT system designed to be used in the diagnostic energy range (20 to 140 keV). In this paper, we demonstrate the use of the MARS scanner, equipped with the Medipix3RX spectroscopic photon-processing detector, to discriminate fat, calcium, and water in tissue. We present data collected from a sample of lamb meat including bone as an illustrative example of human tissue imaging. The data is analyzed using our 3D Algebraic Reconstruction Algorithm (MARS-ART) and by material decomposition based on a constrained linear least squares algorithm. The results presented here clearly show the quantification of lipid-like, water-like and bone-like components of tissue. However, it is also clear to us that better algorithms could extract more information of clinical interest from our data. Because we are one of the first to present data from multi-energy photon-processing small animal CT systems, we make the raw, partial and fully processed data available with the intention that others can analyze it using their familiar routines. The raw, partially processed and fully processed data of lamb tissue along with the phantom calibration data can be found at [http://hdl.handle.net/10092/8531].
△ Less
Submitted 23 January, 2014; v1 submitted 18 November, 2013;
originally announced November 2013.
