-
Vision-Based Multimodal Interfaces: A Survey and Taxonomy for Enhanced Context-Aware System Design
Authors:
Yongquan 'Owen' Hu,
Jingyu Tang,
Xinya Gong,
Zhongyi Zhou,
Shuning Zhang,
Don Samitha Elvitigala,
Florian 'Floyd' Mueller,
Wen Hu,
Aaron J. Quigley
Abstract:
The recent surge in artificial intelligence, particularly in multimodal processing technology, has advanced human-computer interaction, by altering how intelligent systems perceive, understand, and respond to contextual information (i.e., context awareness). Despite such advancements, there is a significant gap in comprehensive reviews examining these advances, especially from a multimodal data pe…
▽ More
The recent surge in artificial intelligence, particularly in multimodal processing technology, has advanced human-computer interaction, by altering how intelligent systems perceive, understand, and respond to contextual information (i.e., context awareness). Despite such advancements, there is a significant gap in comprehensive reviews examining these advances, especially from a multimodal data perspective, which is crucial for refining system design. This paper addresses a key aspect of this gap by conducting a systematic survey of data modality-driven Vision-based Multimodal Interfaces (VMIs). VMIs are essential for integrating multimodal data, enabling more precise interpretation of user intentions and complex interactions across physical and digital environments. Unlike previous task- or scenario-driven surveys, this study highlights the critical role of the visual modality in processing contextual information and facilitating multimodal interaction. Adopting a design framework moving from the whole to the details and back, it classifies VMIs across dimensions, providing insights for developing effective, context-aware systems.
△ Less
Submitted 17 March, 2025; v1 submitted 23 January, 2025;
originally announced January 2025.
-
GustosonicSense: Towards understanding the design of playful gustosonic eating experiences
Authors:
Yan Wang,
Humphrey O. Obie,
Zhuying Li,
Flora D. Salim,
John Grundy,
Florian 'Floyd' Mueller
Abstract:
The pleasure that often comes with eating can be further enhanced with intelligent technology, as the field of human-food interaction suggests. However, knowledge on how to design such pleasure-supporting eating systems is limited. To begin filling this knowledge gap, we designed "GustosonicSense", a novel gustosonic eating system that utilizes wireless earbuds for sensing different eating and dri…
▽ More
The pleasure that often comes with eating can be further enhanced with intelligent technology, as the field of human-food interaction suggests. However, knowledge on how to design such pleasure-supporting eating systems is limited. To begin filling this knowledge gap, we designed "GustosonicSense", a novel gustosonic eating system that utilizes wireless earbuds for sensing different eating and drinking actions with a machine learning algorithm and trigger playful sounds as a way to facilitate pleasurable eating experiences. We present the findings from our design and a study that revealed how we can support the "stimulation", "hedonism", and "reflexivity" for playful human-food interactions. Ultimately, with our work, we aim to support interaction designers in facilitating playful experiences with food.
△ Less
Submitted 16 March, 2024;
originally announced March 2024.
-
Fused Spectatorship: Designing Bodily Experiences Where Spectators Become Players
Authors:
Rakesh Patibanda,
Aryan Saini,
Nathalie Overdevest,
Maria F. Montoya,
Xiang Li,
Yuzheng Chen,
Shreyas Nisal,
Josh Andres,
Jarrod Knibbe,
Elise van den Hoven,
Florian 'Floyd' Mueller
Abstract:
Spectating digital games can be exciting. However, due to its vicarious nature, spectators often wish to engage in the gameplay beyond just watching and cheering. To blur the boundaries between spectators and players, we propose a novel approach called "Fused Spectatorship", where spectators watch their hands play games by loaning bodily control to a computational Electrical Muscle Stimulation (EM…
▽ More
Spectating digital games can be exciting. However, due to its vicarious nature, spectators often wish to engage in the gameplay beyond just watching and cheering. To blur the boundaries between spectators and players, we propose a novel approach called "Fused Spectatorship", where spectators watch their hands play games by loaning bodily control to a computational Electrical Muscle Stimulation (EMS) system. To showcase this concept, we designed three games where spectators loan control over both their hands to the EMS system and watch them play these competitive and collaborative games. A study with 12 participants suggested that participants could not distinguish if they were watching their hands play, or if they were playing the games themselves. We used our results to articulate four spectator experience themes and four fused spectator types, the behaviours they elicited and offer one design consideration to support each of these behaviours. We also discuss the ethical design considerations of our approach to help game designers create future fused spectatorship experiences.
△ Less
Submitted 20 September, 2023; v1 submitted 20 July, 2023;
originally announced July 2023.
-
Auto-Paízo Games: Towards Understanding the Design of Games that Aim to Unify a Player's Physical Body and the Virtual World
Authors:
Rakesh Patibanda,
Chris Hill,
Aryan Saini,
Xiang Li,
Yuzheng Chen,
Andrii Matviienko,
Jarrod Knibbe,
Elise van den Hoven,
Florian 'Floyd' Mueller
Abstract:
Most digital bodily games focus on the body as they use movement as input. However, they also draw the player's focus away from the body as the output occurs on visual displays, creating a divide between the physical body and the virtual world. We propose a novel approach - the "Body as a Play Material" - where a player uses their body as both input and output to unify the physical body and the vi…
▽ More
Most digital bodily games focus on the body as they use movement as input. However, they also draw the player's focus away from the body as the output occurs on visual displays, creating a divide between the physical body and the virtual world. We propose a novel approach - the "Body as a Play Material" - where a player uses their body as both input and output to unify the physical body and the virtual world. To showcase this approach, we designed three games where a player uses one of their hands (input) to play against the other hand (output) by loaning control over its movements to an Electrical Muscle Stimulation (EMS) system. We conducted a thematic analysis on the data obtained from a field study with 12 participants to articulate four player experience themes. We discuss our results about how participants appreciated the engagement with the variety of bodily movements for play and the ambiguity of using their body as a play material. Ultimately, our work aims to unify the physical body and the virtual world.
△ Less
Submitted 20 September, 2023; v1 submitted 20 July, 2023;
originally announced July 2023.
-
Myopic Bike and Say Hi: Games for Empathizing with The Myopic
Authors:
Xiang Li,
Xiaohang Tang,
Xin Tong,
Rakesh Patibanda,
Florian 'Floyd' Mueller,
Hai-Ning Liang
Abstract:
Myopia is an eye condition that makes it difficult for people to focus on faraway objects. It has become one of the most serious eye conditions worldwide and negatively impacts the quality of life of those who suffer from it. Although myopia is prevalent, many non-myopic people have misconceptions about it and encounter challenges empathizing with myopia situations and those who suffer from it. In…
▽ More
Myopia is an eye condition that makes it difficult for people to focus on faraway objects. It has become one of the most serious eye conditions worldwide and negatively impacts the quality of life of those who suffer from it. Although myopia is prevalent, many non-myopic people have misconceptions about it and encounter challenges empathizing with myopia situations and those who suffer from it. In this research, we developed two virtual reality (VR) games, (1) Myopic Bike and (2) Say Hi, to provide a means for the non-myopic population to experience the frustration and difficulties of myopic people. Our two games simulate two inconvenient daily life scenarios (riding a bicycle and greeting someone on the street) that myopic people encounter when not wearing glasses. We evaluated four participants' game experiences through questionnaires and semi-structured interviews. Overall, our two VR games can create an engaging and non-judgmental experience for the non-myopic population to better understand and empathize with those who suffer from myopia.
△ Less
Submitted 28 September, 2021; v1 submitted 11 September, 2021;
originally announced September 2021.
-
vrCAPTCHA: Exploring CAPTCHA Designs in Virtual Reality
Authors:
Xiang Li,
Yuzheng Chen,
Rakesh Patibanda,
Florian 'Floyd' Mueller
Abstract:
With the popularity of online access in virtual reality (VR) devices, it will become important to investigate exclusive and interactive CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) designs for VR devices. In this paper, we first present four traditional two-dimensional (2D) CAPTCHAs (i.e., text-based, image-rotated, image-puzzled, and image-selected CAPTCHAs…
▽ More
With the popularity of online access in virtual reality (VR) devices, it will become important to investigate exclusive and interactive CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) designs for VR devices. In this paper, we first present four traditional two-dimensional (2D) CAPTCHAs (i.e., text-based, image-rotated, image-puzzled, and image-selected CAPTCHAs) in VR. Then, based on the three-dimensional (3D) interaction characteristics of VR devices, we propose two vrCAPTCHA design prototypes (i.e., task-driven and bodily motion-based CAPTCHAs). We conducted a user study with six participants for exploring the feasibility of our two vrCAPTCHAs and traditional CAPTCHAs in VR. We believe that our two vrCAPTCHAs can be an inspiration for the further design of CAPTCHAs in VR.
△ Less
Submitted 25 February, 2021; v1 submitted 24 February, 2021;
originally announced February 2021.
