-
Nested sampling for physical scientists
Authors:
Greg Ashton,
Noam Bernstein,
Johannes Buchner,
Xi Chen,
Gábor Csányi,
Andrew Fowlie,
Farhan Feroz,
Matthew Griffiths,
Will Handley,
Michael Habeck,
Edward Higson,
Michael Hobson,
Anthony Lasenby,
David Parkinson,
Livia B. Pártay,
Matthew Pitkin,
Doris Schneider,
Joshua S. Speagle,
Leah South,
John Veitch,
Philipp Wacker,
David J. Wales,
David Yallup
Abstract:
We review Skilling's nested sampling (NS) algorithm for Bayesian inference and more broadly multi-dimensional integration. After recapitulating the principles of NS, we survey developments in implementing efficient NS algorithms in practice in high-dimensions, including methods for sampling from the so-called constrained prior. We outline the ways in which NS may be applied and describe the applic…
▽ More
We review Skilling's nested sampling (NS) algorithm for Bayesian inference and more broadly multi-dimensional integration. After recapitulating the principles of NS, we survey developments in implementing efficient NS algorithms in practice in high-dimensions, including methods for sampling from the so-called constrained prior. We outline the ways in which NS may be applied and describe the application of NS in three scientific fields in which the algorithm has proved to be useful: cosmology, gravitational-wave astronomy, and materials science. We close by making recommendations for best practice when using NS and by summarizing potential limitations and optimizations of NS.
△ Less
Submitted 31 May, 2022;
originally announced May 2022.
-
Bayesian posterior repartitioning for nested sampling
Authors:
Xi Chen,
Farhan Feroz,
Michael Hobson
Abstract:
Priors in Bayesian analyses often encode informative domain knowledge that can be useful in making the inference process more efficient. Occasionally, however, priors may be unrepresentative of the parameter values for a given dataset, which can result in inefficient parameter space exploration, or even incorrect inferences, particularly for nested sampling (NS) algorithms. Simply broadening the p…
▽ More
Priors in Bayesian analyses often encode informative domain knowledge that can be useful in making the inference process more efficient. Occasionally, however, priors may be unrepresentative of the parameter values for a given dataset, which can result in inefficient parameter space exploration, or even incorrect inferences, particularly for nested sampling (NS) algorithms. Simply broadening the prior in such cases may be inappropriate or impossible in some applications. Hence our previous solution to this problem, known as posterior repartitioning (PR), redefines the prior and likelihood while keeping their product fixed, so that the posterior inferences and evidence estimates remain unchanged, but the efficiency of the NS process is significantly increased. In its most practical form, PR raises the prior to some power beta, which is introduced as an auxiliary variable that must be determined on a case-by-case basis, usually by lowering beta from unity according to some pre-defined `annealing schedule' until the resulting inferences converge to a consistent solution. Here we present a very simple yet powerful alternative Bayesian approach, in which beta is instead treated as a hyperparameter that is inferred from the data alongside the original parameters of the problem, and then marginalised over to obtain the final inference. We show through numerical examples that this Bayesian PR (BPR) method provides a very robust, self-adapting and computationally efficient `hands-off' solution to the problem of unrepresentative priors in Bayesian inference using NS. Moreover, unlike the original PR method, we show that even for representative priors BPR has a negligible computational overhead relative to standard nesting sampling, which suggests that it should be used as the default in all NS analyses.
△ Less
Submitted 4 July, 2022; v1 submitted 13 August, 2019;
originally announced August 2019.
-
Exploring Multi-Modal Distributions with Nested Sampling
Authors:
F. Feroz,
J. Skilling
Abstract:
In performing a Bayesian analysis, two difficult problems often emerge. First, in estimating the parameters of some model for the data, the resulting posterior distribution may be multi-modal or exhibit pronounced (curving) degeneracies. Secondly, in selecting between a set of competing models, calculation of the Bayesian evidence for each model is computationally expensive using existing methods…
▽ More
In performing a Bayesian analysis, two difficult problems often emerge. First, in estimating the parameters of some model for the data, the resulting posterior distribution may be multi-modal or exhibit pronounced (curving) degeneracies. Secondly, in selecting between a set of competing models, calculation of the Bayesian evidence for each model is computationally expensive using existing methods such as thermodynamic integration. Nested Sampling is a Monte Carlo method targeted at the efficient calculation of the evidence, but also produces posterior inferences as a by-product and therefore provides means to carry out parameter estimation as well as model selection. The main challenge in implementing Nested Sampling is to sample from a constrained probability distribution. One possible solution to this problem is provided by the Galilean Monte Carlo (GMC) algorithm. We show results of applying Nested Sampling with GMC to some problems which have proven very difficult for standard Markov Chain Monte Carlo (MCMC) and down-hill methods, due to the presence of large number of local minima and/or pronounced (curving) degeneracies between the parameters. We also discuss the use of Nested Sampling with GMC in Bayesian object detection problems, which are inherently multi-modal and require the evaluation of Bayesian evidence for distinguishing between true and spurious detections.
△ Less
Submitted 19 December, 2013;
originally announced December 2013.
-
SKYNET: an efficient and robust neural network training tool for machine learning in astronomy
Authors:
Philip Graff,
Farhan Feroz,
Michael P. Hobson,
Anthony N. Lasenby
Abstract:
We present the first public release of our generic neural network training algorithm, called SkyNet. This efficient and robust machine learning tool is able to train large and deep feed-forward neural networks, including autoencoders, for use in a wide range of supervised and unsupervised learning applications, such as regression, classification, density estimation, clustering and dimensionality r…
▽ More
We present the first public release of our generic neural network training algorithm, called SkyNet. This efficient and robust machine learning tool is able to train large and deep feed-forward neural networks, including autoencoders, for use in a wide range of supervised and unsupervised learning applications, such as regression, classification, density estimation, clustering and dimensionality reduction. SkyNet uses a `pre-training' method to obtain a set of network parameters that has empirically been shown to be close to a good solution, followed by further optimisation using a regularised variant of Newton's method, where the level of regularisation is determined and adjusted automatically; the latter uses second-order derivative information to improve convergence, but without the need to evaluate or store the full Hessian matrix, by using a fast approximate method to calculate Hessian-vector products. This combination of methods allows for the training of complicated networks that are difficult to optimise using standard backpropagation techniques. SkyNet employs convergence criteria that naturally prevent overfitting, and also includes a fast algorithm for estimating the accuracy of network outputs. The utility and flexibility of SkyNet are demonstrated by application to a number of toy problems, and to astronomical problems focusing on the recovery of structure from blurred and noisy images, the identification of gamma-ray bursters, and the compression and denoising of galaxy images. The SkyNet software, which is implemented in standard ANSI C and fully parallelised using MPI, is available at http://www.mrao.cam.ac.uk/software/skynet/.
△ Less
Submitted 27 January, 2014; v1 submitted 3 September, 2013;
originally announced September 2013.
-
Importance Nested Sampling and the MultiNest Algorithm
Authors:
F. Feroz,
M. P. Hobson,
E. Cameron,
A. N. Pettitt
Abstract:
Bayesian inference involves two main computational challenges. First, in estimating the parameters of some model for the data, the posterior distribution may well be highly multi-modal: a regime in which the convergence to stationarity of traditional Markov Chain Monte Carlo (MCMC) techniques becomes incredibly slow. Second, in selecting between a set of competing models the necessary estimation o…
▽ More
Bayesian inference involves two main computational challenges. First, in estimating the parameters of some model for the data, the posterior distribution may well be highly multi-modal: a regime in which the convergence to stationarity of traditional Markov Chain Monte Carlo (MCMC) techniques becomes incredibly slow. Second, in selecting between a set of competing models the necessary estimation of the Bayesian evidence for each is, by definition, a (possibly high-dimensional) integration over the entire parameter space; again this can be a daunting computational task, although new Monte Carlo (MC) integration algorithms offer solutions of ever increasing efficiency. Nested sampling (NS) is one such contemporary MC strategy targeted at calculation of the Bayesian evidence, but which also enables posterior inference as a by-product, thereby allowing simultaneous parameter estimation and model selection. The widely-used MultiNest algorithm presents a particularly efficient implementation of the NS technique for multi-modal posteriors. In this paper we discuss importance nested sampling (INS), an alternative summation of the MultiNest draws, which can calculate the Bayesian evidence at up to an order of magnitude higher accuracy than `vanilla' NS with no change in the way MultiNest explores the parameter space. This is accomplished by treating as a (pseudo-)importance sample the totality of points collected by MultiNest, including those previously discarded under the constrained likelihood sampling of the NS algorithm. We apply this technique to several challenging test problems and compare the accuracy of Bayesian evidences obtained with INS against those from vanilla NS.
△ Less
Submitted 26 November, 2019; v1 submitted 10 June, 2013;
originally announced June 2013.
-
BAMBI: blind accelerated multimodal Bayesian inference
Authors:
Philip Graff,
Farhan Feroz,
Michael P. Hobson,
Anthony Lasenby
Abstract:
In this paper we present an algorithm for rapid Bayesian analysis that combines the benefits of nested sampling and artificial neural networks. The blind accelerated multimodal Bayesian inference (BAMBI) algorithm implements the MultiNest package for nested sampling as well as the training of an artificial neural network (NN) to learn the likelihood function. In the case of computationally expensi…
▽ More
In this paper we present an algorithm for rapid Bayesian analysis that combines the benefits of nested sampling and artificial neural networks. The blind accelerated multimodal Bayesian inference (BAMBI) algorithm implements the MultiNest package for nested sampling as well as the training of an artificial neural network (NN) to learn the likelihood function. In the case of computationally expensive likelihoods, this allows the substitution of a much more rapid approximation in order to increase significantly the speed of the analysis. We begin by demonstrating, with a few toy examples, the ability of a NN to learn complicated likelihood surfaces. BAMBI's ability to decrease running time for Bayesian inference is then demonstrated in the context of estimating cosmological parameters from Wilkinson Microwave Anisotropy Probe and other observations. We show that valuable speed increases are achieved in addition to obtaining NNs trained on the likelihood functions for the different model and data combinations. These NNs can then be used for an even faster follow-up analysis using the same likelihood and different priors. This is a fully general algorithm that can be applied, without any pre-processing, to other problems with computationally expensive likelihood functions.
△ Less
Submitted 17 February, 2012; v1 submitted 13 October, 2011;
originally announced October 2011.
