Collective inference of the truth of propositions from crowd probability judgments
Authors:
Patrick Stinson,
Jasper van den Bosch,
Trenton Jerde,
Nikolaus Kriegeskorte
Abstract:
Every day, we judge the probability of propositions. When we communicate graded confidence (e.g. "I am 90% sure"), we enable others to gauge how much weight to attach to our judgment. Ideally, people should share their judgments to reach more accurate conclusions collectively. Peer-to-peer tools for collective inference could help debunk disinformation and amplify reliable information on social ne…
▽ More
Every day, we judge the probability of propositions. When we communicate graded confidence (e.g. "I am 90% sure"), we enable others to gauge how much weight to attach to our judgment. Ideally, people should share their judgments to reach more accurate conclusions collectively. Peer-to-peer tools for collective inference could help debunk disinformation and amplify reliable information on social networks, improving democratic discourse. However, individuals fall short of the ideal of well-calibrated probability judgments, and group dynamics can amplify errors and polarize opinions. Here, we connect insights from cognitive science, structured expert judgment, and crowdsourcing to infer the truth of propositions from human probability judgments. In an online experiment, 376 participants judged the probability of each of 1,200 general-knowledge claims for which we have ground truth (451,200 ratings). Aggregating binary judgments by majority vote already exhibits the "wisdom of the crowd"--the superior accuracy of collective inferences relative to individual inferences. However, using continuous probability ratings and accounting for individual accuracy and calibration significantly improves collective inferences. Peer judgment behavior can be modeled probabilistically, and individual parameters capturing each peer's accuracy and miscalibration can be inferred jointly with the claim probabilities. This unsupervised approach can be complemented by supervised methods relying on truth labels to learn models that achieve well-calibrated collective inference. The algorithms we introduce can empower groups of collaborators and online communities to pool their distributed intelligence and jointly judge the probability of propositions with a well-calibrated sense of uncertainty.
△ Less
Submitted 9 January, 2025;
originally announced January 2025.
A shotgun sampling solution for the common input problem in neural connectivity inference
Authors:
Daniel Soudry,
Suraj Keshri,
Patrick Stinson,
Min-hwan Oh,
Garud Iyengar,
Liam Paninski
Abstract:
Inferring connectivity in neuronal networks remains a key challenge in statistical neuroscience. The `common input' problem presents the major roadblock: it is difficult to reliably distinguish causal connections between pairs of observed neurons from correlations induced by common input from unobserved neurons. Since available recording techniques allow us to sample from only a small fraction of…
▽ More
Inferring connectivity in neuronal networks remains a key challenge in statistical neuroscience. The `common input' problem presents the major roadblock: it is difficult to reliably distinguish causal connections between pairs of observed neurons from correlations induced by common input from unobserved neurons. Since available recording techniques allow us to sample from only a small fraction of large networks simultaneously with sufficient temporal resolution, naive connectivity estimators that neglect these common input effects are highly biased. This work proposes a `shotgun' experimental design, in which we observe multiple sub-networks briefly, in a serial manner. Thus, while the full network cannot be observed simultaneously at any given time, we may be able to observe most of it during the entire experiment. Using a generalized linear model for a spiking recurrent neural network, we develop scalable approximate Bayesian methods to perform network inference given this type of data, in which only a small fraction of the network is observed in each time bin. We demonstrate in simulation that, using this method: (1) The shotgun experimental design can eliminate the biases induced by common input effects. (2) Networks with thousands of neurons, in which only a small fraction of the neurons is observed in each time bin, could be quickly and accurately estimated. (3) Performance can be improved if we exploit prior information about the probability of having a connection between two neurons, its dependence on neuronal cell types (e.g., Dale's law), or its dependence on the distance between neurons.
△ Less
Submitted 17 December, 2014; v1 submitted 15 September, 2013;
originally announced September 2013.
