-
The Case for Evaluating Causal Models Using Interventional Measures and Empirical Data
Authors:
Amanda Gentzel,
Dan Garant,
David Jensen
Abstract:
Causal inference is central to many areas of artificial intelligence, including complex reasoning, planning, knowledge-base construction, robotics, explanation, and fairness. An active community of researchers develops and enhances algorithms that learn causal models from data, and this work has produced a series of impressive technical advances. However, evaluation techniques for causal modeling…
▽ More
Causal inference is central to many areas of artificial intelligence, including complex reasoning, planning, knowledge-base construction, robotics, explanation, and fairness. An active community of researchers develops and enhances algorithms that learn causal models from data, and this work has produced a series of impressive technical advances. However, evaluation techniques for causal modeling algorithms have remained somewhat primitive, limiting what we can learn from experimental studies of algorithm performance, constraining the types of algorithms and model representations that researchers consider, and creating a gap between theory and practice. We argue for more frequent use of evaluation techniques that examine interventional measures rather than structural or observational measures, and that evaluate those measures on empirical data rather than synthetic data. We survey the current practice in evaluation and show that the techniques we recommend are rarely used in practice. We show that such techniques are feasible and that data sets are available to conduct such evaluations. We also show that these techniques produce substantially different results than using structural measures and synthetic data.
△ Less
Submitted 1 November, 2019; v1 submitted 11 October, 2019;
originally announced October 2019.
-
Context-Based Concurrent Experience Sharing in Multiagent Systems
Authors:
Dan Garant,
Bruno da Silva,
Victor Lesser,
Chongjie Zhang
Abstract:
One of the key challenges for multi-agent learning is scalability. In this paper, we introduce a technique for speeding up multi-agent learning by exploiting concurrent and incremental experience sharing. This solution adaptively identifies opportunities to transfer experiences between agents and allows for the rapid acquisition of appropriate policies in large-scale, stochastic, homogeneous multi…
▽ More
One of the key challenges for multi-agent learning is scalability. In this paper, we introduce a technique for speeding up multi-agent learning by exploiting concurrent and incremental experience sharing. This solution adaptively identifies opportunities to transfer experiences between agents and allows for the rapid acquisition of appropriate policies in large-scale, stochastic, homogeneous multi-agent systems. We introduce an online, distributed, supervisor-directed transfer technique for constructing high-level characterizations of an agent's dynamic learning environment---called contexts---which are used to identify groups of agents operating under approximately similar dynamics within a short temporal window. A set of supervisory agents computes contextual information for groups of subordinate agents, thereby identifying candidates for experience sharing. Our method uses a tiered architecture to propagate, with low communication overhead, state, action, and reward data amongst the members of each dynamically-identified information-sharing group. We applied this method to a large-scale distributed task allocation problem with hundreds of information-sharing agents operating in an unknown, non-stationary environment. We demonstrate that our approach results in significant performance gains, that it is robust to noise-corrupted or suboptimal context features, and that communication costs scale linearly with the supervisor-to-subordinate ratio.
△ Less
Submitted 6 March, 2017;
originally announced March 2017.
-
Evaluating Causal Models by Comparing Interventional Distributions
Authors:
Dan Garant,
David Jensen
Abstract:
The predominant method for evaluating the quality of causal models is to measure the graphical accuracy of the learned model structure. We present an alternative method for evaluating causal models that directly measures the accuracy of estimated interventional distributions. We contrast such distributional measures with structural measures, such as structural Hamming distance and structural inter…
▽ More
The predominant method for evaluating the quality of causal models is to measure the graphical accuracy of the learned model structure. We present an alternative method for evaluating causal models that directly measures the accuracy of estimated interventional distributions. We contrast such distributional measures with structural measures, such as structural Hamming distance and structural intervention distance, showing that structural measures often correspond poorly to the accuracy of estimated interventional distributions. We use a number of real and synthetic datasets to illustrate various scenarios in which structural measures provide misleading results with respect to algorithm selection and parameter tuning, and we recommend that distributional measures become the new standard for evaluating causal models.
△ Less
Submitted 16 August, 2016;
originally announced August 2016.
