-
Performance and Metacognition Disconnect when Reasoning in Human-AI Interaction
Authors:
Daniela Fernandes,
Steeven Villa,
Salla Nicholls,
Otso Haavisto,
Daniel Buschek,
Albrecht Schmidt,
Thomas Kosch,
Chenxinran Shen,
Robin Welsch
Abstract:
Optimizing human-AI interaction requires users to reflect on their own performance critically. Our paper examines whether people using AI to complete tasks can accurately monitor how well they perform. In Study 1, participants (N = 246) used AI to solve 20 logical problems from the Law School Admission Test. While their task performance improved by three points compared to a norm population, parti…
▽ More
Optimizing human-AI interaction requires users to reflect on their own performance critically. Our paper examines whether people using AI to complete tasks can accurately monitor how well they perform. In Study 1, participants (N = 246) used AI to solve 20 logical problems from the Law School Admission Test. While their task performance improved by three points compared to a norm population, participants overestimated their performance by four points. Interestingly, higher AI literacy was linked to less accurate self-assessment. Participants with more technical knowledge of AI were more confident but less precise in judging their own performance. Using a computational model, we explored individual differences in metacognitive accuracy and found that the Dunning-Kruger effect, usually observed in this task, ceased to exist with AI. Study 2 (N = 452) replicates these findings. We discuss how AI levels metacognitive performance and consider consequences of performance overestimation for interactive AI systems enhancing cognition.
△ Less
Submitted 5 January, 2025; v1 submitted 25 September, 2024;
originally announced September 2024.
-
How To Save A World: The Go-Along Interview as Game Preservation Methodology in Wurm Online
Authors:
Florence Smith Nicholls,
Michael Cook
Abstract:
Massively multiplayer online (MMO) games boomed in the late 1990s to 2000s. In parallel, ethnographic studies of these communities emerged, generally involving participant observation and interviews. Several decades on, many MMOs have been reconfigured, remastered or are potentially no longer accessible at all, which presents challenges for their continued study and long-term preservation. In this…
▽ More
Massively multiplayer online (MMO) games boomed in the late 1990s to 2000s. In parallel, ethnographic studies of these communities emerged, generally involving participant observation and interviews. Several decades on, many MMOs have been reconfigured, remastered or are potentially no longer accessible at all, which presents challenges for their continued study and long-term preservation. In this paper we explore the "go-along" methodology, in which a researcher joins a participant on a walk through a familiar place and asks them questions, as a qualitative research method applicable for the study and preservation of games culture. Though the methodology has been introduced in digital media studies, to date it has had limited application in digital games, if at all. We report on a pilot study exploring applications of the go-along method to the sandbox MMO Wurm Online; a persistent, player-directed world with a rich history. We report on our motivations for the work, our analysis of the resulting interviews, and our reflections on both the use of go-alongs in digital games, as well as the unique and inspiring culture and community of this lesser-known game.
△ Less
Submitted 16 May, 2024;
originally announced May 2024.
-
Permalife Of The Archive: Archaeogaming As Queergaming
Authors:
Florence Smith Nicholls
Abstract:
Archaeogaming and queer games studies have both grown as paradigms in the last decade. The former broadly refers to the archaeological study of games, while the later concerns the application of queer theory to the medium. To date, there has been limited engagement of archaeogamers with queer games scholarship, and vice versa. This article argues that there are epistomological parallels between th…
▽ More
Archaeogaming and queer games studies have both grown as paradigms in the last decade. The former broadly refers to the archaeological study of games, while the later concerns the application of queer theory to the medium. To date, there has been limited engagement of archaeogamers with queer games scholarship, and vice versa. This article argues that there are epistomological parallels between the two; as they are both concerned with the limits and ethics of representation, the personal and political contexts of game development and engagement with video games through transgressive play.
The paper is structured around an extended literature review and three vignettes that reflect on the author's personal experience of conducting archaeogaming research, an ethnographic study of Wurm Online, an archaeological survey of Elden Ring and a player study of the generative archaeology game Nothing Beside Remains. While archaeogaming can learn from the centring of subjective lived experience and labour in the queer games sphere, archaeogaming as a form of game preservation can also benefit queer games studies.
△ Less
Submitted 8 May, 2024;
originally announced May 2024.
-
'That Darned Sandstorm': A Study of Procedural Generation through Archaeological Storytelling
Authors:
Florence Smith Nicholls,
Michael Cook
Abstract:
Procedural content generation has been applied to many domains, especially level design, but the narrative affordances of generated game environments are comparatively understudied. In this paper we present our first attempt to study these effects through the lens of what we call a generative archaeology game that prompts the player to archaeologically interpret the generated content of the game w…
▽ More
Procedural content generation has been applied to many domains, especially level design, but the narrative affordances of generated game environments are comparatively understudied. In this paper we present our first attempt to study these effects through the lens of what we call a generative archaeology game that prompts the player to archaeologically interpret the generated content of the game world. We report on a survey that gathered qualitative and quantitative data on the experiences of 187 participants playing the game Nothing Beside Remains. We provide some preliminary analysis of our intentional attempt to prompt player interpretation, and the unintentional effects of a glitch on the player experience of the game.
△ Less
Submitted 17 April, 2023;
originally announced April 2023.
-
The Dark Souls of Archaeology: Recording Elden Ring
Authors:
Florence Smith Nicholls,
Michael Cook
Abstract:
Archaeology can be broadly defined as the study and interpretation of the past through material remains. Videogame worlds, though immaterial in nature, can also afford opportunities to study the people who existed within them based on what they leave behind. In this paper we present the first formal archaeological survey of a predominantly single-player game, by examining the player-generated cont…
▽ More
Archaeology can be broadly defined as the study and interpretation of the past through material remains. Videogame worlds, though immaterial in nature, can also afford opportunities to study the people who existed within them based on what they leave behind. In this paper we present the first formal archaeological survey of a predominantly single-player game, by examining the player-generated content that is asynchronously distributed to players in the videogame Elden Ring.
△ Less
Submitted 22 April, 2022;
originally announced April 2022.
