-
How is model-related uncertainty quantified and reported in different disciplines?
Authors:
Emily G. Simmonds,
Kwaku Peprah Adjei,
Christoffer Wold Andersen,
Janne Cathrin Hetle Aspheim,
Claudia Battistin,
Nicola Bulso,
Hannah Christensen,
Benjamin Cretois,
Ryan Cubero,
Ivan A. Davidovich,
Lisa Dickel,
Benjamin Dunn,
Etienne Dunn-Sigouin,
Karin Dyrstad,
Sigurd Einum,
Donata Giglio,
Haakon Gjerlow,
Amelie Godefroidt,
Ricardo Gonzalez-Gil,
Soledad Gonzalo Cogno,
Fabian Grosse,
Paul Halloran,
Mari F. Jensen,
John James Kennedy,
Peter Egge Langsaether
, et al. (18 additional authors not shown)
Abstract:
How do we know how much we know? Quantifying uncertainty associated with our modelling work is the only way we can answer how much we know about any phenomenon. With quantitative science now highly influential in the public sphere and the results from models translating into action, we must support our conclusions with sufficient rigour to produce useful, reproducible results. Incomplete considera…
▽ More
How do we know how much we know? Quantifying uncertainty associated with our modelling work is the only way we can answer how much we know about any phenomenon. With quantitative science now highly influential in the public sphere and the results from models translating into action, we must support our conclusions with sufficient rigour to produce useful, reproducible results. Incomplete consideration of model-based uncertainties can lead to false conclusions with real world impacts. Despite these potentially damaging consequences, uncertainty consideration is incomplete both within and across scientific fields. We take a unique interdisciplinary approach and conduct a systematic audit of model-related uncertainty quantification from seven scientific fields, spanning the biological, physical, and social sciences. Our results show no single field is achieving complete consideration of model uncertainties, but together we can fill the gaps. We propose opportunities to improve the quantification of uncertainty through use of a source framework for uncertainty consideration, model type specific guidelines, improved presentation, and shared best practice. We also identify shared outstanding challenges (uncertainty in input data, balancing trade-offs, error propagation, and defining how much uncertainty is required). Finally, we make nine concrete recommendations for current practice (following good practice guidelines and an uncertainty checklist, presenting uncertainty numerically, and propagating model-related uncertainty into conclusions), future research priorities (uncertainty in input data, quantifying uncertainty in complex models, and the importance of missing uncertainty in different contexts), and general research standards across the sciences (transparency about study limitations and dedicated uncertainty sections of manuscripts).
△ Less
Submitted 1 July, 2022; v1 submitted 24 June, 2022;
originally announced June 2022.
-
Restricted Boltzmann Machines as Models of Interacting Variables
Authors:
Nicola Bulso,
Yasser Roudi
Abstract:
We study the type of distributions that Restricted Boltzmann Machines (RBMs) with different activation functions can express by investigating the effect of the activation function of the hidden nodes on the marginal distribution they impose on observed binary nodes. We report an exact expression for these marginals in the form of a model of interacting binary variables with the explicit form of th…
▽ More
We study the type of distributions that Restricted Boltzmann Machines (RBMs) with different activation functions can express by investigating the effect of the activation function of the hidden nodes on the marginal distribution they impose on observed binary nodes. We report an exact expression for these marginals in the form of a model of interacting binary variables with the explicit form of the interactions depending on the hidden node activation function. We study the properties of these interactions in detail and evaluate how the accuracy with which the RBM approximates distributions over binary variables depends on the hidden node activation function and on the number of hidden nodes. When the inferred RBM parameters are weak, an intuitive pattern is found for the expression of the interaction terms which reduces substantially the differences across activation functions. We show that the weak parameter approximation is a good approximation for different RBMs trained on the MNIST dataset. Interestingly, in these cases, the mapping reveals that the inferred models are essentially low order interaction models.
△ Less
Submitted 29 March, 2021;
originally announced March 2021.
-
On the complexity of logistic regression models
Authors:
Nicola Bulso,
Matteo Marsili,
Yasser Roudi
Abstract:
We investigate the complexity of logistic regression models which is defined by counting the number of indistinguishable distributions that the model can represent (Balasubramanian, 1997). We find that the complexity of logistic models with binary inputs does not only depend on the number of parameters but also on the distribution of inputs in a non-trivial way which standard treatments of complex…
▽ More
We investigate the complexity of logistic regression models which is defined by counting the number of indistinguishable distributions that the model can represent (Balasubramanian, 1997). We find that the complexity of logistic models with binary inputs does not only depend on the number of parameters but also on the distribution of inputs in a non-trivial way which standard treatments of complexity do not address. In particular, we observe that correlations among inputs induce effective dependencies among parameters thus constraining the model and, consequently, reducing its complexity. We derive simple relations for the upper and lower bounds of the complexity. Furthermore, we show analytically that, defining the model parameters on a finite support rather than the entire axis, decreases the complexity in a manner that critically depends on the size of the domain. Based on our findings, we propose a novel model selection criterion which takes into account the entropy of the input distribution. We test our proposal on the problem of selecting the input variables of a logistic regression model in a Bayesian Model Selection framework. In our numerical tests, we find that, while the reconstruction errors of standard model selection approaches (AIC, BIC, $\ell_1$ regularization) strongly depend on the sparsity of the ground truth, the reconstruction error of our method is always close to the minimum in all conditions of sparsity, data size and strength of input correlations. Finally, we observe that, when considering categorical instead of binary inputs, in a simple and mathematically tractable case, the contribution of the alphabet size to the complexity is very small compared to that of parameter space dimension. We further explore the issue by analysing the dataset of the "13 keys to the White House" which is a method for forecasting the outcomes of US presidential elections.
△ Less
Submitted 1 March, 2019;
originally announced March 2019.
-
Sparse model selection in the highly under-sampled regime
Authors:
Nicola Bulso,
Matteo Marsili,
Yasser Roudi
Abstract:
We propose a method for recovering the structure of a sparse undirected graphical model when very few samples are available. The method decides about the presence or absence of bonds between pairs of variable by considering one pair at a time and using a closed form formula, analytically derived by calculating the posterior probability for every possible model explaining a two body system using Je…
▽ More
We propose a method for recovering the structure of a sparse undirected graphical model when very few samples are available. The method decides about the presence or absence of bonds between pairs of variable by considering one pair at a time and using a closed form formula, analytically derived by calculating the posterior probability for every possible model explaining a two body system using Jeffreys prior. The approach does not rely on the optimisation of any cost functions and consequently is much faster than existing algorithms. Despite this time and computational advantage, numerical results show that for several sparse topologies the algorithm is comparable to the best existing algorithms, and is more accurate in the presence of hidden variables. We apply this approach to the analysis of US stock market data and to neural data, in order to show its efficiency in recovering robust statistical dependencies in real data with non stationary correlations in time and space.
△ Less
Submitted 2 January, 2017; v1 submitted 2 March, 2016;
originally announced March 2016.
-
Effective dissipation and nonlocality induced by nonparaxiality
Authors:
Nicola Bulso,
Claudio Conti
Abstract:
We investigate beam diffraction and spatial modulation instability of coherent light beams propagating in the non-paraxial regime in a nonlinear Kerr medium. We study the instability of plane wave solutions in terms of the degree of non-paraxiality, beyond the regime of validity of the nonlinear Schroedinger equation. We also numerically analyze the way non-paraxial terms breaks the integrability…
▽ More
We investigate beam diffraction and spatial modulation instability of coherent light beams propagating in the non-paraxial regime in a nonlinear Kerr medium. We study the instability of plane wave solutions in terms of the degree of non-paraxiality, beyond the regime of validity of the nonlinear Schroedinger equation. We also numerically analyze the way non-paraxial terms breaks the integrability and affect the periodical evolution of higher order soliton solutions. We discuss non-locality and effective dissipation introduced by the non-linear coupling with evanescent waves.
△ Less
Submitted 5 February, 2014;
originally announced February 2014.
