-
Replication Markets: Results, Lessons, Challenges and Opportunities in AI Replication
Authors:
Yang Liu,
Michael Gordon,
Juntao Wang,
Michael Bishop,
Yiling Chen,
Thomas Pfeiffer,
Charles Twardy,
Domenico Viganola
Abstract:
The last decade saw the emergence of systematic large-scale replication projects in the social and behavioral sciences, (Camerer et al., 2016, 2018; Ebersole et al., 2016; Klein et al., 2014, 2018; Collaboration, 2015). These projects were driven by theoretical and conceptual concerns about a high fraction of "false positives" in the scientific publications (Ioannidis, 2005) (and a high prevalence…
▽ More
The last decade saw the emergence of systematic large-scale replication projects in the social and behavioral sciences, (Camerer et al., 2016, 2018; Ebersole et al., 2016; Klein et al., 2014, 2018; Collaboration, 2015). These projects were driven by theoretical and conceptual concerns about a high fraction of "false positives" in the scientific publications (Ioannidis, 2005) (and a high prevalence of "questionable research practices" (Simmons, Nelson, and Simonsohn, 2011). Concerns about the credibility of research findings are not unique to the behavioral and social sciences; within Computer Science, Artificial Intelligence (AI) and Machine Learning (ML) are areas of particular concern (Lucic et al., 2018; Freire, Bonnet, and Shasha, 2012; Gundersen and Kjensmo, 2018; Henderson et al., 2018). Given the pioneering role of the behavioral and social sciences in the promotion of novel methodologies to improve the credibility of research, it is a promising approach to analyze the lessons learned from this field and adjust strategies for Computer Science, AI and ML In this paper, we review approaches used in the behavioral and social sciences and in the DARPA SCORE project. We particularly focus on the role of human forecasting of replication outcomes, and how forecasting can leverage the information gained from relatively labor and resource-intensive replications. We will discuss opportunities and challenges of using these approaches to monitor and improve the credibility of research areas in Computer Science, AI, and ML.
△ Less
Submitted 9 May, 2020;
originally announced May 2020.
-
Interval Elicitation of Forecasts in a Prediction Market Reveals Lack of Anchoring "Bias"
Authors:
Kenneth Olson,
Kathryn Laskey,
Charles Twardy
Abstract:
In an online prediction market, forecasters who could not see the current state of the market until they made their own separate estimates moved their estimates closer to the market forecast when the current state of the market became known. Their first edits to the market forecast were very similar to the first edits of forecasters who could always see the current state of the market, and forecas…
▽ More
In an online prediction market, forecasters who could not see the current state of the market until they made their own separate estimates moved their estimates closer to the market forecast when the current state of the market became known. Their first edits to the market forecast were very similar to the first edits of forecasters who could always see the current state of the market, and forecasters in both conditions had similar accuracy. These results suggest that our more elaborate forecast elicitation method might not improve forecasts and that any anchoring on the state of the market does not constitute an error in judgment.
△ Less
Submitted 29 June, 2014;
originally announced June 2014.
-
Trade-based Asset Model using Dynamic Junction Tree for Combinatorial Prediction Markets
Authors:
Wei Sun,
Kathryn Laskey,
Charles Twardy,
Robin Hanson,
Brandon Goldfedder
Abstract:
Prediction markets have demonstrated their value for aggregating collective expertise. Combinatorial prediction markets allow forecasts not only on base events, but also on conditional and/or Boolean combinations of events. We describe a trade-based combinatorial prediction market asset management system, called Dynamic Asset Cluster (DAC), that improves both time and space efficiency over the met…
▽ More
Prediction markets have demonstrated their value for aggregating collective expertise. Combinatorial prediction markets allow forecasts not only on base events, but also on conditional and/or Boolean combinations of events. We describe a trade-based combinatorial prediction market asset management system, called Dynamic Asset Cluster (DAC), that improves both time and space efficiency over the method of, which maintains parallel junction trees for assets and probabilities. The basic data structure is the asset block, which compactly represents a set of trades made by a user. A user's asset model consists of a set of asset blocks representing the user's entire trade history. A junction tree is created dynamically from the asset blocks to compute a user's minimum and expected assets.
△ Less
Submitted 29 June, 2014;
originally announced June 2014.
-
Probability and Asset Updating using Bayesian Networks for Combinatorial Prediction Markets
Authors:
Wei Sun,
Robin Hanson,
Kathryn Blackmond Laskey,
Charles Twardy
Abstract:
A market-maker-based prediction market lets forecasters aggregate information by editing a consensus probability distribution either directly or by trading securities that pay off contingent on an event of interest. Combinatorial prediction markets allow trading on any event that can be specified as a combination of a base set of events. However, explicitly representing the full joint distribution…
▽ More
A market-maker-based prediction market lets forecasters aggregate information by editing a consensus probability distribution either directly or by trading securities that pay off contingent on an event of interest. Combinatorial prediction markets allow trading on any event that can be specified as a combination of a base set of events. However, explicitly representing the full joint distribution is infeasible for markets with more than a few base events. A factored representation such as a Bayesian network (BN) can achieve tractable computation for problems with many related variables. Standard BN inference algorithms, such as the junction tree algorithm, can be used to update a representation of the entire joint distribution given a change to any local conditional probability. However, in order to let traders reuse assets from prior trades while never allowing assets to become negative, a BN based prediction market also needs to update a representation of each user's assets and find the conditional state in which a user has minimum assets. Users also find it useful to see their expected assets given an edit outcome. We show how to generalize the junction tree algorithm to perform all these computations.
△ Less
Submitted 16 October, 2012;
originally announced October 2012.
