-
A Principled Approach to Bayesian Transfer Learning
Authors:
Adam Bretherton,
Joshua J. Bon,
David J. Warne,
Kerrie Mengersen,
Christopher Drovandi
Abstract:
Updating $\textit{a priori}$ information given some observed data is the core tenet of Bayesian inference. Bayesian transfer learning extends this idea by incorporating information from a related dataset to improve the inference on the observed data which may have been collected under slightly different settings. The use of related information can be useful when the observed data is scarce, for ex…
▽ More
Updating $\textit{a priori}$ information given some observed data is the core tenet of Bayesian inference. Bayesian transfer learning extends this idea by incorporating information from a related dataset to improve the inference on the observed data which may have been collected under slightly different settings. The use of related information can be useful when the observed data is scarce, for example. Current Bayesian transfer learning methods that are based on the so-called $\textit{power prior}$ can adaptively transfer information from related data. Unfortunately, it is not always clear under which scenario Bayesian transfer learning performs best or even if it will improve Bayesian inference. Additionally, current power prior methods rely on conjugacy to evaluate the posterior of interest. We propose using leave-one-out cross validation on the target dataset as a means of evaluating Bayesian transfer learning methods. Further, we introduce a new framework, $\textit{transfer sequential Monte Carlo}$, for power prior approaches that efficiently chooses the transfer parameter while avoiding the need for conjugate priors. We assess the performance of our proposed methods in two comprehensive simulation studies.
△ Less
Submitted 19 March, 2025; v1 submitted 27 February, 2025;
originally announced February 2025.
-
Federated Learning for Non-factorizable Models using Deep Generative Prior Approximations
Authors:
Conor Hassan,
Joshua J Bon,
Elizaveta Semenova,
Antonietta Mira,
Kerrie Mengersen
Abstract:
Federated learning (FL) allows for collaborative model training across decentralized clients while preserving privacy by avoiding data sharing. However, current FL methods assume conditional independence between client models, limiting the use of priors that capture dependence, such as Gaussian processes (GPs). We introduce the Structured Independence via deep Generative Model Approximation (SIGMA…
▽ More
Federated learning (FL) allows for collaborative model training across decentralized clients while preserving privacy by avoiding data sharing. However, current FL methods assume conditional independence between client models, limiting the use of priors that capture dependence, such as Gaussian processes (GPs). We introduce the Structured Independence via deep Generative Model Approximation (SIGMA) prior which enables FL for non-factorizable models across clients, expanding the applicability of FL to fields such as spatial statistics, epidemiology, environmental science, and other domains where modeling dependencies is crucial. The SIGMA prior is a pre-trained deep generative model that approximates the desired prior and induces a specified conditional independence structure in the latent variables, creating an approximate model suitable for FL settings. We demonstrate the SIGMA prior's effectiveness on synthetic data and showcase its utility in a real-world example of FL for spatial data, using a conditional autoregressive prior to model spatial dependence across Australia. Our work enables new FL applications in domains where modeling dependent data is essential for accurate predictions and decision-making.
△ Less
Submitted 25 May, 2024;
originally announced May 2024.
-
Being Bayesian in the 2020s: opportunities and challenges in the practice of modern applied Bayesian statistics
Authors:
Joshua J. Bon,
Adam Bretherton,
Katie Buchhorn,
Susanna Cramb,
Christopher Drovandi,
Conor Hassan,
Adrianne L. Jenner,
Helen J. Mayfield,
James M. McGree,
Kerrie Mengersen,
Aiden Price,
Robert Salomone,
Edgar Santos-Fernandez,
Julie Vercelloni,
Xiaoyu Wang
Abstract:
Building on a strong foundation of philosophy, theory, methods and computation over the past three decades, Bayesian approaches are now an integral part of the toolkit for most statisticians and data scientists. Whether they are dedicated Bayesians or opportunistic users, applied professionals can now reap many of the benefits afforded by the Bayesian paradigm. In this paper, we touch on six moder…
▽ More
Building on a strong foundation of philosophy, theory, methods and computation over the past three decades, Bayesian approaches are now an integral part of the toolkit for most statisticians and data scientists. Whether they are dedicated Bayesians or opportunistic users, applied professionals can now reap many of the benefits afforded by the Bayesian paradigm. In this paper, we touch on six modern opportunities and challenges in applied Bayesian statistics: intelligent data collection, new data sources, federated analysis, inference for implicit models, model transfer and purposeful software products.
△ Less
Submitted 17 January, 2023; v1 submitted 17 November, 2022;
originally announced November 2022.
-
Bayesian score calibration for approximate models
Authors:
Joshua J Bon,
David J Warne,
David J Nott,
Christopher Drovandi
Abstract:
Scientists continue to develop increasingly complex mechanistic models to reflect their knowledge more realistically. Statistical inference using these models can be challenging since the corresponding likelihood function is often intractable and model simulation may be computationally burdensome. Fortunately, in many of these situations, it is possible to adopt a surrogate model or approximate li…
▽ More
Scientists continue to develop increasingly complex mechanistic models to reflect their knowledge more realistically. Statistical inference using these models can be challenging since the corresponding likelihood function is often intractable and model simulation may be computationally burdensome. Fortunately, in many of these situations, it is possible to adopt a surrogate model or approximate likelihood function. It may be convenient to conduct Bayesian inference directly with the surrogate, but this can result in bias and poor uncertainty quantification. In this paper we propose a new method for adjusting approximate posterior samples to reduce bias and produce more accurate uncertainty quantification. We do this by optimizing a transform of the approximate posterior that maximizes a scoring rule. Our approach requires only a (fixed) small number of complex model simulations and is numerically stable. We demonstrate good performance of the new method on several examples of increasing complexity.
△ Less
Submitted 27 October, 2023; v1 submitted 10 November, 2022;
originally announced November 2022.
-
Monte Carlo twisting for particle filters
Authors:
Joshua J Bon,
Christopher Drovandi,
Anthony Lee
Abstract:
We consider the problem of designing efficient particle filters for twisted Feynman--Kac models. Particle filters using twisted models can deliver low error approximations of statistical quantities and such twisting functions can be learnt iteratively. Practical implementations of these algorithms are complicated by the need to (i) sample from the twisted transition dynamics, and (ii) calculate th…
▽ More
We consider the problem of designing efficient particle filters for twisted Feynman--Kac models. Particle filters using twisted models can deliver low error approximations of statistical quantities and such twisting functions can be learnt iteratively. Practical implementations of these algorithms are complicated by the need to (i) sample from the twisted transition dynamics, and (ii) calculate the twisted potential functions. We expand the class of applicable models using rejection sampling for (i) and unbiased approximations for (ii) using a random weight particle filter. We characterise the average acceptance rates within the particle filter in order to control the computational cost, and analyse the asymptotic variance. Empirical results show the mean squared error of the normalising constant estimate in our method is smaller than a memory-equivalent particle filter but not a computation-equivalent filter. Both comparisons are improved when more efficient sampling is possible which we demonstrate on a stochastic volatility model.
△ Less
Submitted 8 August, 2022;
originally announced August 2022.
-
Accelerating sequential Monte Carlo with surrogate likelihoods
Authors:
Joshua J Bon,
Anthony Lee,
Christopher Drovandi
Abstract:
Delayed-acceptance is a technique for reducing computational effort for Bayesian models with expensive likelihoods. Using a delayed-acceptance kernel for Markov chain Monte Carlo can reduce the number of expensive likelihoods evaluations required to approximate a posterior expectation. Delayed-acceptance uses a surrogate, or approximate, likelihood to avoid evaluation of the expensive likelihood w…
▽ More
Delayed-acceptance is a technique for reducing computational effort for Bayesian models with expensive likelihoods. Using a delayed-acceptance kernel for Markov chain Monte Carlo can reduce the number of expensive likelihoods evaluations required to approximate a posterior expectation. Delayed-acceptance uses a surrogate, or approximate, likelihood to avoid evaluation of the expensive likelihood when possible. Within the sequential Monte Carlo framework, we utilise the history of the sampler to adaptively tune the surrogate likelihood to yield better approximations of the expensive likelihood, and use a surrogate first annealing schedule to further increase computational efficiency. Moreover, we propose a framework for optimising computation time whilst avoiding particle degeneracy, which encapsulates existing strategies in the literature. Overall, we develop a novel algorithm for computationally efficient SMC with expensive likelihood functions. The method is applied to static Bayesian models, which we demonstrate on toy and real examples, code for which is available at https://github.com/bonStats/smcdar.
△ Less
Submitted 20 July, 2021; v1 submitted 8 September, 2020;
originally announced September 2020.
-
Polling bias and undecided voter allocations: US Presidential elections, 2004 - 2016
Authors:
Joshua J Bon,
Timothy Ballard,
Bernard Baffour
Abstract:
Accounting for undecided and uncertain voters is a challenging issue for predicting election results from public opinion polls. Undecided voters typify the uncertainty of swing voters in polls but are often ignored or allocated to each candidate in a simple, deterministic manner. Historically this may have been adequate because the undecided were comparatively small enough to assume that they do n…
▽ More
Accounting for undecided and uncertain voters is a challenging issue for predicting election results from public opinion polls. Undecided voters typify the uncertainty of swing voters in polls but are often ignored or allocated to each candidate in a simple, deterministic manner. Historically this may have been adequate because the undecided were comparatively small enough to assume that they do not affect the relative proportions of the decided voters. However, in the presence of high numbers of undecided voters, these static rules may in fact bias election predictions from election poll authors and meta-poll analysts. In this paper, we examine the effect of undecided voters in the 2016 US presidential election to the previous three presidential elections. We show there were a relatively high number of undecided voters over the campaign and on election day, and that the allocation of undecided voters in this election was not consistent with two-party proportional (or even) allocations. We find evidence that static allocation regimes are inadequate for election prediction models and that probabilistic allocations may be superior. We also estimate the bias attributable to polling agencies, often referred to as "house effects".
△ Less
Submitted 16 January, 2019; v1 submitted 28 March, 2017;
originally announced March 2017.
