-
How to Measure Human-AI Prediction Accuracy in Explainable AI Systems
Authors:
Sujay Koujalgi,
Andrew Anderson,
Iyadunni Adenuga,
Shikha Soneji,
Rupika Dikkala,
Teresita Guzman Nader,
Leo Soccio,
Sourav Panda,
Rupak Kumar Das,
Margaret Burnett,
Jonathan Dodge
Abstract:
Assessing an AI system's behavior-particularly in Explainable AI Systems-is sometimes done empirically, by measuring people's abilities to predict the agent's next move-but how to perform such measurements? In empirical studies with humans, an obvious approach is to frame the task as binary (i.e., prediction is either right or wrong), but this does not scale. As output spaces increase, so do floor…
▽ More
Assessing an AI system's behavior-particularly in Explainable AI Systems-is sometimes done empirically, by measuring people's abilities to predict the agent's next move-but how to perform such measurements? In empirical studies with humans, an obvious approach is to frame the task as binary (i.e., prediction is either right or wrong), but this does not scale. As output spaces increase, so do floor effects, because the ratio of right answers to wrong answers quickly becomes very small. The crux of the problem is that the binary framing is failing to capture the nuances of the different degrees of "wrongness." To address this, we begin by proposing three mathematical bases upon which to measure "partial wrongness." We then uses these bases to perform two analyses on sequential decision-making domains: the first is an in-lab study with 86 participants on a size-36 action space; the second is a re-analysis of a prior study on a size-4 action space. Other researchers adopting our operationalization of the prediction task and analysis methodology will improve the rigor of user studies conducted with that task, which is particularly important when the domain features a large output space.
△ Less
Submitted 23 August, 2024;
originally announced September 2024.
-
Understanding Driver Agency in RideSharing
Authors:
Iyadunni Adenuga,
Benjamin Hanrahan
Abstract:
Agency is an important human characteristic that users of automated complex technologies are usually denied. This affects the user's experience leading to decreased satisfaction and productivity. In this paper, we consider the ridesharing context and interviewed 7 drivers to understand the controls that would improve the agency they feel. The results show that they desire transparency, community a…
▽ More
Agency is an important human characteristic that users of automated complex technologies are usually denied. This affects the user's experience leading to decreased satisfaction and productivity. In this paper, we consider the ridesharing context and interviewed 7 drivers to understand the controls that would improve the agency they feel. The results show that they desire transparency, community and an effective ability to seek redress.
△ Less
Submitted 30 December, 2023;
originally announced January 2024.
-
Conceptualizing the Relationship between AI Explanations and User Agency
Authors:
Iyadunni Adenuga,
Jonathan Dodge
Abstract:
We grapple with the question: How, for whom and why should explainable artificial intelligence (XAI) aim to support the user goal of agency? In particular, we analyze the relationship between agency and explanations through a user-centric lens through case studies and thought experiments. We find that explanation serves as one of several possible first steps for agency by allowing the user convert…
▽ More
We grapple with the question: How, for whom and why should explainable artificial intelligence (XAI) aim to support the user goal of agency? In particular, we analyze the relationship between agency and explanations through a user-centric lens through case studies and thought experiments. We find that explanation serves as one of several possible first steps for agency by allowing the user convert forethought to outcome in a more effective manner in future interactions. Also, we observe that XAI systems might better cater to laypersons, particularly "tinkerers", when combining explanations and user control, so they can make meaningful changes.
△ Less
Submitted 5 December, 2023;
originally announced December 2023.
-
Lowering Barriers to Application Development With Cloud-Native Domain-Specific Functions
Authors:
José Miguel Pérez-Álvarez,
Adrian Mos,
Benjamin V. Hanrahan,
Iyadunni J. Adenuga
Abstract:
Creating and maintaining a modern, heterogeneous set of client applications remains an obstacle for many businesses and individuals. While simple domain-specific graphical languages and libraries can empower a variety of users to create application behaviors and logic, using these languages to produce and maintain a set of heterogeneous client applications is a challenge. Primarily because each cl…
▽ More
Creating and maintaining a modern, heterogeneous set of client applications remains an obstacle for many businesses and individuals. While simple domain-specific graphical languages and libraries can empower a variety of users to create application behaviors and logic, using these languages to produce and maintain a set of heterogeneous client applications is a challenge. Primarily because each client typically requires the developers to both understand and embed the domain-specific logic. This is because application logic must be encoded to some extent in both the server and client sides.
In this paper, we propose an alternative approach, which allows the specification of application logic to reside solely on the cloud. We have built a system where reusable application components can be assembled on the cloud in different logical chains and the client is largely decoupled from this logic and is solely concerned with how data is displayed and gathered from users of the application. In this way, the chaining of requests and responses is done by the cloud and the client side has no knowledge of the application logic.
An additional effect of our approach is that the client side developer is able to immediately see any changes they make, while executing the logic residing on the cloud. This further allows more novice programmers to perform these customizations, as they do not need to `get the full application working' and are able to see the results of their code as they go, thereby lowering the obstacles to businesses and individuals to produce and maintain applications. Furthermore, this decoupling enables the quick generation and customization of a variety of application clients, ranging from web to mobile devices and personal assistants, while customizing one or more as needed.
△ Less
Submitted 20 April, 2022;
originally announced April 2022.
