-
Bias in studies of prenatal exposures using real-world data due to pregnancy identification method
Authors:
Chase D. Latour,
Jessie K. Edwards,
Michele Jonsson Funk,
Elizabeth A. Suarez,
Kim Boggess,
Mollie E. Wood
Abstract:
Background: Researchers typically identify pregnancies in healthcare data based on observed outcomes (e.g., delivery). This outcome-based approach misses pregnancies that received prenatal care but whose outcomes were not recorded (e.g., at-home miscarriage), potentially inducing selection bias in effect estimates for prenatal exposures. Alternatively, prenatal encounters can be used to identify p…
▽ More
Background: Researchers typically identify pregnancies in healthcare data based on observed outcomes (e.g., delivery). This outcome-based approach misses pregnancies that received prenatal care but whose outcomes were not recorded (e.g., at-home miscarriage), potentially inducing selection bias in effect estimates for prenatal exposures. Alternatively, prenatal encounters can be used to identify pregnancies, including those with unobserved outcomes. However, this prenatal approach requires methods to address missing data. Methods: We simulated 10,000,000 pregnancies and estimated the total effect of initiating treatment on the risk of preeclampsia. We generated data for 36 scenarios in which we varied the effect of treatment on miscarriage and/or preeclampsia; the percentage with missing outcomes (5% or 20%); and the cause of missingness: (1) measured covariates, (2) unobserved miscarriage, and (3) a mix of both. We then created three analytic samples to address missing pregnancy outcomes: observed deliveries, observed deliveries and miscarriages, and all pregnancies. Treatment effects were estimated using non-parametric direct standardization. Results: Risk differences (RDs) and risk ratios (RRs) from the three analytic samples were similarly biased when all missingness was due to unobserved miscarriage (log-transformed RR bias range: -0.12-0.33 among observed deliveries; -0.11-0.32 among observed deliveries and miscarriages; and -0.11-0.32 among all pregnancies). When predictors of missingness were measured, only the all pregnancies approach was unbiased (-0.27-0.33; -0.29-0.03; and -0.02-0.01, respectively). Conclusions: When all missingness was due to miscarriage, the analytic samples returned similar effect estimates. Only among all pregnancies did bias decrease as the proportion of missingness due to measured variables increased.
△ Less
Submitted 16 April, 2025;
originally announced April 2025.
-
Evaluating Human Trust in LLM-Based Planners: A Preliminary Study
Authors:
Shenghui Chen,
Yunhao Yang,
Kayla Boggess,
Seongkook Heo,
Lu Feng,
Ufuk Topcu
Abstract:
Large Language Models (LLMs) are increasingly used for planning tasks, offering unique capabilities not found in classical planners such as generating explanations and iterative refinement. However, trust--a critical factor in the adoption of planning systems--remains underexplored in the context of LLM-based planning tasks. This study bridges this gap by comparing human trust in LLM-based planner…
▽ More
Large Language Models (LLMs) are increasingly used for planning tasks, offering unique capabilities not found in classical planners such as generating explanations and iterative refinement. However, trust--a critical factor in the adoption of planning systems--remains underexplored in the context of LLM-based planning tasks. This study bridges this gap by comparing human trust in LLM-based planners with classical planners through a user study in a Planning Domain Definition Language (PDDL) domain. Combining subjective measures, such as trust questionnaires, with objective metrics like evaluation accuracy, our findings reveal that correctness is the primary driver of trust and performance. Explanations provided by the LLM improved evaluation accuracy but had limited impact on trust, while plan refinement showed potential for increasing trust without significantly enhancing evaluation accuracy.
△ Less
Submitted 27 February, 2025;
originally announced February 2025.
-
Explainable Multi-Agent Reinforcement Learning for Temporal Queries
Authors:
Kayla Boggess,
Sarit Kraus,
Lu Feng
Abstract:
As multi-agent reinforcement learning (MARL) systems are increasingly deployed throughout society, it is imperative yet challenging for users to understand the emergent behaviors of MARL agents in complex environments. This work presents an approach for generating policy-level contrastive explanations for MARL to answer a temporal user query, which specifies a sequence of tasks completed by agents…
▽ More
As multi-agent reinforcement learning (MARL) systems are increasingly deployed throughout society, it is imperative yet challenging for users to understand the emergent behaviors of MARL agents in complex environments. This work presents an approach for generating policy-level contrastive explanations for MARL to answer a temporal user query, which specifies a sequence of tasks completed by agents with possible cooperation. The proposed approach encodes the temporal query as a PCTL logic formula and checks if the query is feasible under a given MARL policy via probabilistic model checking. Such explanations can help reconcile discrepancies between the actual and anticipated multi-agent behaviors. The proposed approach also generates correct and complete explanations to pinpoint reasons that make a user query infeasible. We have successfully applied the proposed approach to four benchmark MARL domains (up to 9 agents in one domain). Moreover, the results of a user study show that the generated explanations significantly improve user performance and satisfaction.
△ Less
Submitted 17 May, 2023;
originally announced May 2023.
-
Toward Policy Explanations for Multi-Agent Reinforcement Learning
Authors:
Kayla Boggess,
Sarit Kraus,
Lu Feng
Abstract:
Advances in multi-agent reinforcement learning (MARL) enable sequential decision making for a range of exciting multi-agent applications such as cooperative AI and autonomous driving. Explaining agent decisions is crucial for improving system transparency, increasing user satisfaction, and facilitating human-agent collaboration. However, existing works on explainable reinforcement learning mostly…
▽ More
Advances in multi-agent reinforcement learning (MARL) enable sequential decision making for a range of exciting multi-agent applications such as cooperative AI and autonomous driving. Explaining agent decisions is crucial for improving system transparency, increasing user satisfaction, and facilitating human-agent collaboration. However, existing works on explainable reinforcement learning mostly focus on the single-agent setting and are not suitable for addressing challenges posed by multi-agent environments. We present novel methods to generate two types of policy explanations for MARL: (i) policy summarization about the agent cooperation and task sequence, and (ii) language explanations to answer queries about agent behavior. Experimental results on three MARL domains demonstrate the scalability of our methods. A user study shows that the generated explanations significantly improve user performance and increase subjective ratings on metrics such as user satisfaction.
△ Less
Submitted 23 May, 2022; v1 submitted 26 April, 2022;
originally announced April 2022.
-
Multi-Objective Controller Synthesis with Uncertain Human Preferences
Authors:
Shenghui Chen,
Kayla Boggess,
David Parker,
Lu Feng
Abstract:
Complex real-world applications of cyber-physical systems give rise to the need for multi-objective controller synthesis, which concerns the problem of computing an optimal controller subject to multiple (possibly conflicting) criteria. The relative importance of objectives is often specified by human decision-makers. However, there is inherent uncertainty in human preferences (e.g., due to artifa…
▽ More
Complex real-world applications of cyber-physical systems give rise to the need for multi-objective controller synthesis, which concerns the problem of computing an optimal controller subject to multiple (possibly conflicting) criteria. The relative importance of objectives is often specified by human decision-makers. However, there is inherent uncertainty in human preferences (e.g., due to artifacts resulting from different preference elicitation methods). In this paper, we formalize the notion of uncertain human preferences and present a novel approach that accounts for this uncertainty in the context of multi-objective controller synthesis for Markov decision processes (MDPs). Our approach is based on mixed-integer linear programming and synthesizes an optimally permissive multi-strategy that satisfies uncertain human preferences with respect to a multi-objective property. Experimental results on a range of large case studies show that the proposed approach is feasible and scalable across varying MDP model sizes and uncertainty levels of human preferences. Evaluation via an online user study also demonstrates the quality and benefits of the synthesized controllers.
△ Less
Submitted 2 March, 2022; v1 submitted 10 May, 2021;
originally announced May 2021.
-
Towards Personalized Explanation of Robot Path Planning via User Feedback
Authors:
Kayla Boggess,
Shenghui Chen,
Lu Feng
Abstract:
Prior studies have found that explaining robot decisions and actions helps to increase system transparency, improve user understanding, and enable effective human-robot collaboration. In this paper, we present a system for generating personalized explanations of robot path planning via user feedback. We consider a robot navigating in an environment modeled as a Markov decision process (MDP), and d…
▽ More
Prior studies have found that explaining robot decisions and actions helps to increase system transparency, improve user understanding, and enable effective human-robot collaboration. In this paper, we present a system for generating personalized explanations of robot path planning via user feedback. We consider a robot navigating in an environment modeled as a Markov decision process (MDP), and develop an algorithm to automatically generate a personalized explanation of an optimal MDP policy, based on the user preference regarding four elements (i.e., objective, locality, specificity, and corpus). In addition, we design the system to interact with users via answering users' further questions about the generated explanations. Users have the option to update their preferences to view different explanations. The system is capable of detecting and resolving any preference conflict via user interaction. The results of an online user study show that the generated personalized explanations improve user satisfaction, while the majority of users liked the system's capabilities of question-answering and conflict detection/resolution.
△ Less
Submitted 5 March, 2021; v1 submitted 1 November, 2020;
originally announced November 2020.
-
Towards Transparent Robotic Planning via Contrastive Explanations
Authors:
Shenghui Chen,
Kayla Boggess,
Lu Feng
Abstract:
Providing explanations of chosen robotic actions can help to increase the transparency of robotic planning and improve users' trust. Social sciences suggest that the best explanations are contrastive, explaining not just why one action is taken, but why one action is taken instead of another. We formalize the notion of contrastive explanations for robotic planning policies based on Markov decision…
▽ More
Providing explanations of chosen robotic actions can help to increase the transparency of robotic planning and improve users' trust. Social sciences suggest that the best explanations are contrastive, explaining not just why one action is taken, but why one action is taken instead of another. We formalize the notion of contrastive explanations for robotic planning policies based on Markov decision processes, drawing on insights from the social sciences. We present methods for the automated generation of contrastive explanations with three key factors: selectiveness, constrictiveness, and responsibility. The results of a user study with 100 participants on the Amazon Mechanical Turk platform show that our generated contrastive explanations can help to increase users' understanding and trust of robotic planning policies while reducing users' cognitive burden.
△ Less
Submitted 16 March, 2020;
originally announced March 2020.
