-
Ordinal regression for meta-analysis of test accuracy: a flexible approach for utilising all threshold data
Authors:
Enzo Cerullo,
Haley E. Jones,
Tim Lucas,
Nicola J. Cooper,
Alex J. Sutton
Abstract:
Standard methods for meta-analysis and network-meta-analysis of test accuracy do not fully utilise available evidence, as they analyse thresholds separately, resulting in a loss of data unless every study reports all thresholds - which rarely occurs. Furthermore, previously proposed "multiple threshold" models introduce different problems: making overly restrictive assumptions, or failing to provi…
▽ More
Standard methods for meta-analysis and network-meta-analysis of test accuracy do not fully utilise available evidence, as they analyse thresholds separately, resulting in a loss of data unless every study reports all thresholds - which rarely occurs. Furthermore, previously proposed "multiple threshold" models introduce different problems: making overly restrictive assumptions, or failing to provide summary sensitivity and specificity estimates across thresholds.
To address this, we proposed a series of ordinal regression-based models, representing a natural extension of established frameworks. Our approach offers notable advantages: (i) complete data utilisation: rather than discarding information like standard methods, we incorporate all threshold data; (ii) threshold-specific inference: by providing summary accuracy estimates across thresholds, our models deliver critical information for clinical decision-making; (iii) enhanced flexibility: unlike previous "multiple thresholds" approaches, our methodology imposes fewer assumptions, leading to better accuracy estimates; (iv) our models use an induced-Dirichlet framework, allowing for either fixed-effects or random-effects cutpoint parameters, whilst also allowing for intuitive cutpoint priors.
Our (ongoing) simulation study - based on real-world anxiety and depression screening data - demonstrates notably better accuracy estimates than previous approaches, even when the number of categories is high.
Furthermore, we implemented these models in a user-friendly R package - MetaOrdDTA (https://github.com/CerulloE1996/MetaOrdDTA). The package uses Stan and produces MCMC summaries, sROC plots with credible/prediction regions, and meta-regression.
Overall, our approach establishes a more comprehensive framework for synthesising test accuracy data, better serving systematic reviewers and clinical decision-makers.
△ Less
Submitted 29 May, 2025;
originally announced May 2025.
-
Uncertainty quantification of neural network models of evolving processes via Langevin sampling
Authors:
Cosmin Safta,
Reese E. Jones,
Ravi G. Patel,
Raelynn Wonnacot,
Dan S. Bolintineanu,
Craig M. Hamel,
Sharlotte L. B. Kramer
Abstract:
We propose a scalable, approximate inference hypernetwork framework for a general model of history-dependent processes. The flexible data model is based on a neural ordinary differential equation (NODE) representing the evolution of internal states together with a trainable observation model subcomponent. The posterior distribution corresponding to the data model parameters (weights and biases) fo…
▽ More
We propose a scalable, approximate inference hypernetwork framework for a general model of history-dependent processes. The flexible data model is based on a neural ordinary differential equation (NODE) representing the evolution of internal states together with a trainable observation model subcomponent. The posterior distribution corresponding to the data model parameters (weights and biases) follows a stochastic differential equation with a drift term related to the score of the posterior that is learned jointly with the data model parameters. This Langevin sampling approach offers flexibility in balancing the computational budget between the evaluation cost of the data model and the approximation of the posterior density of its parameters. We demonstrate performance of the ensemble sampling hypernetwork on chemical reaction and material physics data and compare it to standard variational inference.
△ Less
Submitted 19 May, 2025; v1 submitted 21 April, 2025;
originally announced April 2025.
-
Condensed Stein Variational Gradient Descent for Uncertainty Quantification of Neural Networks
Authors:
Govinda Anantha Padmanabha,
Cosmin Safta,
Nikolaos Bouklas,
Reese E. Jones
Abstract:
We propose a Stein variational gradient descent method to concurrently sparsify, train, and provide uncertainty quantification of a complexly parameterized model such as a neural network. It employs a graph reconciliation and condensation process to reduce complexity and increase similarity in the Stein ensemble of parameterizations. Therefore, the proposed condensed Stein variational gradient (cS…
▽ More
We propose a Stein variational gradient descent method to concurrently sparsify, train, and provide uncertainty quantification of a complexly parameterized model such as a neural network. It employs a graph reconciliation and condensation process to reduce complexity and increase similarity in the Stein ensemble of parameterizations. Therefore, the proposed condensed Stein variational gradient (cSVGD) method provides uncertainty quantification on parameters, not just outputs. Furthermore, the parameter reduction speeds up the convergence of the Stein gradient descent as it reduces the combinatorial complexity by aligning and differentiating the sensitivity to parameters. These properties are demonstrated with an illustrative example and an application to a representation problem in solid mechanics.
△ Less
Submitted 20 December, 2024;
originally announced December 2024.
-
Exploring the impact of gamification on engagement in a statistics classroom
Authors:
Eilidh Jack,
Craig Alexander,
Elinor M Jones
Abstract:
In recent years, the integration of gamification into educational settings has garnered significant attention as a means to enhance student engagement and learning outcomes. By leveraging gamified elements such as points and leaderboards, educators aim to promote active participation, motivation, and deeper understanding among students. This study investigates the effect of gamification on student…
▽ More
In recent years, the integration of gamification into educational settings has garnered significant attention as a means to enhance student engagement and learning outcomes. By leveraging gamified elements such as points and leaderboards, educators aim to promote active participation, motivation, and deeper understanding among students. This study investigates the effect of gamification on student engagement in a flipped statistics classroom environment. The findings suggest that gamification strategies, when effectively implemented, can have a positive impact on student motivation and engagement. This paper concludes with recommendations for educators, potential challenges such as superficial engagement and demotivation, and future directions for research to address these challenges and further explore the potential of gamification in fostering student success.
△ Less
Submitted 22 October, 2024; v1 submitted 28 February, 2024;
originally announced February 2024.
-
Channelling Multimodality Through a Unimodalizing Transport: Warp-U Sampler and Stochastic Bridge Sampling Estimator
Authors:
Fei Ding,
Shiyuan He,
David E. Jones,
Xiao-Li Meng
Abstract:
Monte Carlo integration is a powerful tool for scientific and statistical computation, but faces significant challenges when the integrand is a multi-modal distribution, even when the mode locations are known. This work introduces novel Monte Carlo sampling and integration estimation strategies for the multi-modal context by leveraging a generalized version of the stochastic Warp-U transformation…
▽ More
Monte Carlo integration is a powerful tool for scientific and statistical computation, but faces significant challenges when the integrand is a multi-modal distribution, even when the mode locations are known. This work introduces novel Monte Carlo sampling and integration estimation strategies for the multi-modal context by leveraging a generalized version of the stochastic Warp-U transformation Wang et al. [2022]. We propose two flexible classes of Warp-U transformations, one based on a general location-scale-skew mixture model and a second using neural ordinary differential equations. We develop an efficient sampling strategy called Warp-U sampling, which applies a Warp-U transformation to map a multi-modal density into a uni-modal one, then inverts the transformation with injected stochasticity. In high dimensions, our approach relies on information about the mode locations, but requires minimal tuning and demonstrates better mixing properties than conventional methods with identical mode information. To improve normalizing constant estimation once samples are obtained, we propose a stochastic Warp-U bridge sampling estimator, which we demonstrate has higher asymptotic precision per CPU second compared to the original approach proposed by Wang et al. [2022]. We also establish the ergodicity of our sampling algorithm. The effectiveness and current limitations of our methods are illustrated through simulation studies and an application to exoplanet detection.
△ Less
Submitted 7 March, 2025; v1 submitted 1 January, 2024;
originally announced January 2024.
-
Bayesian Optimal Experimental Design for Constitutive Model Calibration
Authors:
Denielle Ricciardi,
Tom Seidl,
Brian Lester,
Amanda Jones,
Elizabeth Jones
Abstract:
Computational simulation is increasingly relied upon for high-consequence engineering decisions, and a foundational element to solid mechanics simulations, such as finite element analysis (FEA), is a credible constitutive or material model. Calibration of these complex models is an essential step; however, the selection, calibration and validation of material models is often a discrete, multi-stag…
▽ More
Computational simulation is increasingly relied upon for high-consequence engineering decisions, and a foundational element to solid mechanics simulations, such as finite element analysis (FEA), is a credible constitutive or material model. Calibration of these complex models is an essential step; however, the selection, calibration and validation of material models is often a discrete, multi-stage process that is decoupled from material characterization activities, which means the data collected does not always align with the data that is needed. To address this issue, an integrated workflow for delivering an enhanced characterization and calibration procedure (Interlaced Characterization and Calibration (ICC)) is introduced. This framework leverages Bayesian optimal experimental design (BOED) to select the optimal load path for a cruciform specimen in order to collect the most informative data for model calibration. The critical first piece of algorithm development is to demonstrate the active experimental design for a fast model with simulated data. For this demonstration, a material point simulator that models a plane stress elastoplastic material subject to bi-axial loading was chosen. The ICC framework is demonstrated on two exemplar problems in which BOED is used to determine which load step to take, e.g., in which direction to increment the strain, at each iteration of the characterization and calibration cycle. Calibration results from data obtained by adaptively selecting the load path within the ICC algorithm are compared to results from data generated under two naive static load paths that were chosen a priori based on human intuition. In these exemplar problems, data generated in an adaptive setting resulted in calibrated model parameters with reduced measures of uncertainty compared to the static settings.
△ Less
Submitted 26 October, 2023; v1 submitted 21 August, 2023;
originally announced August 2023.
-
Variable selection in balance regression with applications to microbiome compositional data
Authors:
Jing Ma,
Paizhe Xie,
Kristyn Pantoja,
David E. Jones
Abstract:
Compositional data, where only relative abundances are available, are common in microbiome and other high-throughput sequencing studies. Log ratios between groups of variables serve as key biomarkers in these settings. However, selecting predictive log ratios is a combinatorial challenge, and existing greedy search-based methods are computationally expensive, limiting their applicability to high-d…
▽ More
Compositional data, where only relative abundances are available, are common in microbiome and other high-throughput sequencing studies. Log ratios between groups of variables serve as key biomarkers in these settings. However, selecting predictive log ratios is a combinatorial challenge, and existing greedy search-based methods are computationally expensive, limiting their applicability to high-dimensional data. We introduce the supervised log ratio (SLR) method, a novel and efficient approach for selecting predictive log ratios in high-dimensional settings. SLR first screens active variables using univariate regression on log ratio transformed data and then applies principal balance analysis to define balance biomarkers. Our approach leverages both the relationship between the response and predictors and the correlations among the predictors to improve accuracy in variable selection and prediction. Through simulations and two case studies -- one on inflammatory bowel disease (IBD) and another on colorectal cancer (CRC) -- we demonstrate that SLR outperforms existing methods, particularly in high-dimensional settings. SLR is implemented in an R package, publicly available on GitHub.
△ Less
Submitted 31 March, 2025; v1 submitted 31 March, 2023;
originally announced April 2023.
-
Practical Guidance for Bayesian Inference in Astronomy
Authors:
Gwendolyn M. Eadie,
Joshua S. Speagle,
Jessi Cisewski-Kehe,
Daniel Foreman-Mackey,
Daniela Huppenkothen,
David E. Jones,
Aaron Springford,
Hyungsuk Tak
Abstract:
In the last two decades, Bayesian inference has become commonplace in astronomy. At the same time, the choice of algorithms, terminology, notation, and interpretation of Bayesian inference varies from one sub-field of astronomy to the next, which can lead to confusion to both those learning and those familiar with Bayesian statistics. Moreover, the choice varies between the astronomy and statistic…
▽ More
In the last two decades, Bayesian inference has become commonplace in astronomy. At the same time, the choice of algorithms, terminology, notation, and interpretation of Bayesian inference varies from one sub-field of astronomy to the next, which can lead to confusion to both those learning and those familiar with Bayesian statistics. Moreover, the choice varies between the astronomy and statistics literature, too. In this paper, our goal is two-fold: (1) provide a reference that consolidates and clarifies terminology and notation across disciplines, and (2) outline practical guidance for Bayesian inference in astronomy. Highlighting both the astronomy and statistics literature, we cover topics such as notation, specification of the likelihood and prior distributions, inference using the posterior distribution, and posterior predictive checking. It is not our intention to introduce the entire field of Bayesian data analysis -- rather, we present a series of useful practices for astronomers who already have an understanding of the Bayesian "nuts and bolts" and wish to increase their expertise and extend their knowledge. Moreover, as the field of astrostatistics and astroinformatics continues to grow, we hope this paper will serve as both a helpful reference and as a jumping off point for deeper dives into the statistics and astrostatistics literature.
△ Less
Submitted 9 February, 2023;
originally announced February 2023.
-
MetaBayesDTA: Codeless Bayesian meta-analysis of test accuracy, with or without a gold standard
Authors:
Enzo Cerullo,
Alex J. Sutton,
Hayley E. Jones,
Olivia Wu,
Terry J. Quinn,
Nicola J. Cooper
Abstract:
Introduction: Despite their applicability, statistical models used for the meta-analysis of test accuracy require specialised knowledge to implement, with the necessary level of expertise having recently increased. This is due to the development and recommendation to use more sophisticated methods; such as those in Version 2 of the Cochrane Handbook for Systematic Reviews of Diagnostic Test Accura…
▽ More
Introduction: Despite their applicability, statistical models used for the meta-analysis of test accuracy require specialised knowledge to implement, with the necessary level of expertise having recently increased. This is due to the development and recommendation to use more sophisticated methods; such as those in Version 2 of the Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. This paper describes a web-based application that extends the functionality of previous applications, making many advanced analysis methods more accessible.
Methods: We sought to create an extended, stand-alone, Bayesian version of MetaDTA, which (i) has the benefits of previously proposed applications and addresses key limitations of them, (ii) is accessible to researchers who do not have the specific expertise required to fit such models, and (iii) is suitable for experienced analysts. We created the application using Shiny and Stan.
Results: We created MetaBayesDTA (https://crsu.shinyapps.io/MetaBayesDTA/), which allows users to conduct meta-analysis of test accuracy, with or without a gold standard. The application addresses several key limitations of other applications. For instance, for the bivariate model, one can conduct subgroup analysis, univariate meta-regression, and comparative test accuracy evaluation. Meanwhile, for the model which does not assume a perfect gold standard, the application can account for the fact that studies use different reference tests.
Conclusions: Due to its user-friendliness and broad array of features, MetaBayesDTA should appeal to a wide variety of researchers. We anticipate that the application will encourage wider use of more advanced methods, which ultimately should improve the quality of test accuracy reviews.
△ Less
Submitted 15 November, 2022; v1 submitted 10 November, 2022;
originally announced November 2022.
-
Bayesian learning of forest and tree graphical models
Authors:
Edmund Jones
Abstract:
In Bayesian learning of Gaussian graphical model structure, it is common to restrict attention to certain classes of graphs and approximate the posterior distribution by repeatedly moving from one graph to another, using MCMC or methods such as stochastic shotgun search (SSS). I give two corrected versions of an algorithm for non-decomposable graphs and discuss random graph distributions, in parti…
▽ More
In Bayesian learning of Gaussian graphical model structure, it is common to restrict attention to certain classes of graphs and approximate the posterior distribution by repeatedly moving from one graph to another, using MCMC or methods such as stochastic shotgun search (SSS). I give two corrected versions of an algorithm for non-decomposable graphs and discuss random graph distributions, in particular as prior distributions. The main topic of the thesis is Bayesian structure-learning with forests or trees. Restricting attention to these graphs can be justified using theorems on random graphs. I describe how to use the Chow$\unicode{x2013}$Liu algorithm and the Matrix Tree Theorem to find the MAP forest and certain quantities in the posterior distribution on trees. I give adapted versions of MCMC and SSS for approximating the posterior distribution for forests and trees, and systems for storing these graphs so that it is easy to choose moves to neighbouring graphs. Experiments show that SSS with trees does well when the true graph is a tree or sparse graph. SSS with trees or forests does better than SSS with decomposable graphs in certain cases. Graph priors improve detection of hubs but need large ranges of probabilities. MCMC on forests fails to mix well and MCMC on trees is slower than SSS. (For a longer abstract see the thesis.)
△ Less
Submitted 31 August, 2021;
originally announced August 2021.
-
eBASCS: Disentangling Overlapping Astronomical Sources II, using Spatial, Spectral, and Temporal Information
Authors:
Antoine D. Meyer,
David A. van Dyk,
Vinay L. Kashyap,
Luis F. Campos,
David E. Jones,
Aneta Siemiginowska,
Andreas Zezas
Abstract:
The analysis of individual X-ray sources that appear in a crowded field can easily be compromised by the misallocation of recorded events to their originating sources. Even with a small number of sources, that nonetheless have overlapping point spread functions, the allocation of events to sources is a complex task that is subject to uncertainty. We develop a Bayesian method designed to sift high-…
▽ More
The analysis of individual X-ray sources that appear in a crowded field can easily be compromised by the misallocation of recorded events to their originating sources. Even with a small number of sources, that nonetheless have overlapping point spread functions, the allocation of events to sources is a complex task that is subject to uncertainty. We develop a Bayesian method designed to sift high-energy photon events from multiple sources with overlapping point spread functions, leveraging the differences in their spatial, spectral, and temporal signatures. The method probabilistically assigns each event to a given source. Such a disentanglement allows more detailed spectral or temporal analysis to focus on the individual component in isolation, free of contamination from other sources or the background. We are also able to compute source parameters of interest like their locations, relative brightness, and background contamination, while accounting for the uncertainty in event assignments. Simulation studies that include event arrival time information demonstrate that the temporal component improves event disambiguation beyond using only spatial and spectral information. The proposed methods correctly allocate up to 65% more events than the corresponding algorithms that ignore event arrival time information. We apply our methods to two stellar X-ray binaries, UV Cet and HBC515 A, observed with Chandra. We demonstrate that our methods are capable of removing the contamination due to a strong flare on UV Cet B in its companion approximately 40 times weaker during that event, and that evidence for spectral variability at timescales of a few ks can be determined in HBC515 Aa and HBC515 Ab.
△ Less
Submitted 18 May, 2021;
originally announced May 2021.
-
Meta-analysis of dichotomous and ordinal tests without a gold standard
Authors:
Enzo Cerullo,
Hayley E. Jones,
Olivia Carter,
Terry J. Quinn,
Nicola J. Cooper,
Alex J. Sutton
Abstract:
Standard methods for the meta-analysis of medical tests without a gold standard are limited to dichotomous data. Multivariate probit models are used to analyze correlated binary data, and can be extended to multivariate ordered probit models to model ordinal data. Within the context of an imperfect gold standard, they have previously been used for the analysis of dichotomous and ordinal tests in a…
▽ More
Standard methods for the meta-analysis of medical tests without a gold standard are limited to dichotomous data. Multivariate probit models are used to analyze correlated binary data, and can be extended to multivariate ordered probit models to model ordinal data. Within the context of an imperfect gold standard, they have previously been used for the analysis of dichotomous and ordinal tests in a single study, and for the meta-analysis of dichotomous tests. In this paper, we developed a hierarchical, latent class multivariate probit model for the simultaneous meta-analysis of ordinal and dichotomous tests without assuming a gold standard. The model can accommodate a hierarchical partial pooling model on the conditional within-study correlations, enabling one to obtain summary estimates of joint test accuracy. Dichotomous tests use probit regression likelihoods and ordinal tests use ordered probit regression likelihoods. We fitted the models using Stan, which uses a state-of-the-art Hamiltonian Monte Carlo algorithm. We applied the models to a dataset in which studies evaluated the accuracy of tests, and test combinations, for deep vein thrombosis. We first demonstrated the issues with dichotomising test accuracy data a priori without a gold standard by fitting models which dichotomised the ordinal test data, and then we applied models which do not dichotomise the data. Furthermore, we fitted and compared a variety of other models, including those which assumed conditional independence and dependence between tests, and those assuming perfect and an imperfect gold standard.
△ Less
Submitted 26 April, 2022; v1 submitted 11 March, 2021;
originally announced March 2021.
-
Selective Classification Can Magnify Disparities Across Groups
Authors:
Erik Jones,
Shiori Sagawa,
Pang Wei Koh,
Ananya Kumar,
Percy Liang
Abstract:
Selective classification, in which models can abstain on uncertain predictions, is a natural approach to improving accuracy in settings where errors are costly but abstentions are manageable. In this paper, we find that while selective classification can improve average accuracies, it can simultaneously magnify existing accuracy disparities between various groups within a population, especially in…
▽ More
Selective classification, in which models can abstain on uncertain predictions, is a natural approach to improving accuracy in settings where errors are costly but abstentions are manageable. In this paper, we find that while selective classification can improve average accuracies, it can simultaneously magnify existing accuracy disparities between various groups within a population, especially in the presence of spurious correlations. We observe this behavior consistently across five vision and NLP datasets. Surprisingly, increasing abstentions can even decrease accuracies on some groups. To better understand this phenomenon, we study the margin distribution, which captures the model's confidences over all predictions. For symmetric margin distributions, we prove that whether selective classification monotonically improves or worsens accuracy is fully determined by the accuracy at full coverage (i.e., without any abstentions) and whether the distribution satisfies a property we call left-log-concavity. Our analysis also shows that selective classification tends to magnify full-coverage accuracy disparities. Motivated by our analysis, we train distributionally-robust models that achieve similar full-coverage accuracies across groups and show that selective classification uniformly improves each group on these models. Altogether, our results suggest that selective classification should be used with care and underscore the importance of training models to perform equally well across groups at full coverage.
△ Less
Submitted 14 April, 2021; v1 submitted 27 October, 2020;
originally announced October 2020.
-
Bayesian optimization for automatic design of face stimuli
Authors:
Pedro F. da Costa,
Romy Lorenz,
Ricardo Pio Monti,
Emily Jones,
Robert Leech
Abstract:
Investigating the cognitive and neural mechanisms involved with face processing is a fundamental task in modern neuroscience and psychology. To date, the majority of such studies have focused on the use of pre-selected stimuli. The absence of personalized stimuli presents a serious limitation as it fails to account for how each individual face processing system is tuned to cultural embeddings or h…
▽ More
Investigating the cognitive and neural mechanisms involved with face processing is a fundamental task in modern neuroscience and psychology. To date, the majority of such studies have focused on the use of pre-selected stimuli. The absence of personalized stimuli presents a serious limitation as it fails to account for how each individual face processing system is tuned to cultural embeddings or how it is disrupted in disease. In this work, we propose a novel framework which combines generative adversarial networks (GANs) with Bayesian optimization to identify individual response patterns to many different faces. Formally, we employ Bayesian optimization to efficiently search the latent space of state-of-the-art GAN models, with the aim to automatically generate novel faces, to maximize an individual subject's response. We present results from a web-based proof-of-principle study, where participants rated images of themselves generated via performing Bayesian optimization over the latent space of a GAN. We show how the algorithm can efficiently locate an individual's optimal face while mapping out their response across different semantic transformations of a face; inter-individual analyses suggest how the approach can provide rich information about individual differences in face processing.
△ Less
Submitted 20 July, 2020;
originally announced July 2020.
-
Functional PCA with Covariate Dependent Mean and Covariance Structure
Authors:
Fei Ding,
Shiyuan He,
David E. Jones,
Jianhua Z. Huang
Abstract:
Incorporating covariates into functional principal component analysis (PCA) can substantially improve the representation efficiency of the principal components and predictive performance. However, many existing functional PCA methods do not make use of covariates, and those that do often have high computational cost or make overly simplistic assumptions that are violated in practice. In this artic…
▽ More
Incorporating covariates into functional principal component analysis (PCA) can substantially improve the representation efficiency of the principal components and predictive performance. However, many existing functional PCA methods do not make use of covariates, and those that do often have high computational cost or make overly simplistic assumptions that are violated in practice. In this article, we propose a new framework, called Covariate Dependent Functional Principal Component Analysis (CD-FPCA), in which both the mean and covariance structure depend on covariates. We propose a corresponding estimation algorithm, which makes use of spline basis representations and roughness penalties, and is substantially more computationally efficient than competing approaches of adequate estimation and prediction accuracy. A key aspect of our work is our novel approach for modeling the covariance function and ensuring that it is symmetric positive semi-definite. We demonstrate the advantages of our methodology through a simulation study and an astronomical data analysis.
△ Less
Submitted 19 August, 2023; v1 submitted 30 January, 2020;
originally announced January 2020.
-
Quantifying Observed Prior Impact
Authors:
David E Jones,
Robert N Trangucci,
Yang Chen
Abstract:
We distinguish two questions (i) how much information does the prior contain? and (ii) what is the effect of the prior? Several measures have been proposed for quantifying effective prior sample size, for example Clarke [1996] and Morita et al. [2008]. However, these measures typically ignore the likelihood for the inference currently at hand, and therefore address (i) rather than (ii). Since in p…
▽ More
We distinguish two questions (i) how much information does the prior contain? and (ii) what is the effect of the prior? Several measures have been proposed for quantifying effective prior sample size, for example Clarke [1996] and Morita et al. [2008]. However, these measures typically ignore the likelihood for the inference currently at hand, and therefore address (i) rather than (ii). Since in practice (ii) is of great concern, Reimherr et al. [2014] introduced a new class of effective prior sample size measures based on prior-likelihood discordance. We take this idea further towards its natural Bayesian conclusion by proposing measures of effective prior sample size that not only incorporate the general mathematical form of the likelihood but also the specific data at hand. Thus, our measures do not average across datasets from the working model, but condition on the current observed data. Consequently, our measures can be highly variable, but we demonstrate that this is because the impact of a prior can be highly variable. Our measures are Bayes estimates of meaningful quantities and well communicate the extent to which inference is determined by the prior, or framed differently, the amount of effort saved due to having prior information. We illustrate our ideas through a number of examples including a Gaussian conjugate model (continuous observations), a Beta-Binomial model (discrete observations), and a linear regression model (two unknown parameters). Future work on further developments of the methodology and an application to astronomy are discussed at the end.
△ Less
Submitted 28 January, 2020;
originally announced January 2020.
-
A review of problem- and team-based methods for teaching statistics in Higher Education
Authors:
Elinor Jones,
Tom Palmer
Abstract:
The teaching of statistics in higher education in the UK is still largely lecture-based. This is despite recommendations such as those given by the American Statistical Association's GAISE report that more emphasis should be placed on active learning strategies where students take more responsibility for their own learning. One possible model is that of collaborative learning, where students learn…
▽ More
The teaching of statistics in higher education in the UK is still largely lecture-based. This is despite recommendations such as those given by the American Statistical Association's GAISE report that more emphasis should be placed on active learning strategies where students take more responsibility for their own learning. One possible model is that of collaborative learning, where students learn in groups through carefully crafted `problems', which has long been suggested as a strategy for teaching statistics.
In this article, we review two specific approaches that fall under the collaborative learning model: problem- and team-based learning. We consider the evidence for changing to this model of teaching in statistics, as well as give practical suggestions on how this could be implemented in typical statistics classes in Higher Education.
△ Less
Submitted 10 March, 2021; v1 submitted 1 October, 2019;
originally announced October 2019.
-
Designing Test Information and Test Information in Design
Authors:
David E. Jones,
Xiao-Li Meng
Abstract:
DeGroot (1962) developed a general framework for constructing Bayesian measures of the expected information that an experiment will provide for estimation. We propose an analogous framework for measures of information for hypothesis testing. In contrast to estimation information measures that are typically used for surface estimation, test information measures are more useful in experimental desig…
▽ More
DeGroot (1962) developed a general framework for constructing Bayesian measures of the expected information that an experiment will provide for estimation. We propose an analogous framework for measures of information for hypothesis testing. In contrast to estimation information measures that are typically used for surface estimation, test information measures are more useful in experimental design for hypothesis testing and model selection. In particular, we obtain a probability based measure, which has more appealing properties than variance based measures in design contexts where decision problems are of interest. The underlying intuition of our design proposals is straightforward: to distinguish between models we should collect data from regions of the covariate space for which the models differ most. Nicolae et al. (2008) gave an asymptotic equivalence between their test information measures and Fisher information. We extend this result to all test information measures under our framework. Simulation studies and an application in astronomy demonstrate the utility of our approach, and provide comparison to other methods including that of Box and Hill (1967).
△ Less
Submitted 16 June, 2019;
originally announced June 2019.
-
Improving Exoplanet Detection Power: Multivariate Gaussian Process Models for Stellar Activity
Authors:
David E. Jones,
David C. Stenning,
Eric B. Ford,
Robert L. Wolpert,
Thomas J. Loredo,
Christian Gilbertson,
Xavier Dumusque
Abstract:
The radial velocity method is one of the most successful techniques for detecting exoplanets. It works by detecting the velocity of a host star induced by the gravitational effect of an orbiting planet, specifically the velocity along our line of sight, which is called the radial velocity of the star. Low-mass planets typically cause their host star to move with radial velocities of 1 m/s or less.…
▽ More
The radial velocity method is one of the most successful techniques for detecting exoplanets. It works by detecting the velocity of a host star induced by the gravitational effect of an orbiting planet, specifically the velocity along our line of sight, which is called the radial velocity of the star. Low-mass planets typically cause their host star to move with radial velocities of 1 m/s or less. By analyzing a time series of stellar spectra from a host star, modern astronomical instruments can in theory detect such planets. However, in practice, intrinsic stellar variability (e.g., star spots, convective motion, pulsations) affects the spectra and often mimics a radial velocity signal. This signal contamination makes it difficult to reliably detect low-mass planets. A principled approach to recovering planet radial velocity signals in the presence of stellar activity was proposed by Rajpaul et al. (2015). It uses a multivariate Gaussian process model to jointly capture time series of the apparent radial velocity and multiple indicators of stellar activity. We build on this work in two ways: (i) we propose using dimension reduction techniques to construct new high-information stellar activity indicators; and (ii) we extend the Rajpaul et al. (2015) model to a larger class of models and use a power-based model comparison procedure to select the best model. Despite significant interest in exoplanets, previous efforts have not performed large-scale stellar activity model selection or attempted to evaluate models based on planet detection power. In the case of main sequence G2V stars, we find that our method substantially improves planet detection power compared to previous state-of-the-art approaches.
△ Less
Submitted 25 August, 2020; v1 submitted 3 November, 2017;
originally announced November 2017.
-
Warp Bridge Sampling: The Next Generation
Authors:
Lazhi Wang,
David E. Jones,
Xiao-Li Meng
Abstract:
Bridge sampling is an effective Monte Carlo method for estimating the ratio of normalizing constants of two probability densities, a routine computational problem in statistics, physics, chemistry, and other fields. The Monte Carlo error of the bridge sampling estimator is determined by the amount of overlap between the two densities. In the case of uni-modal densities, Warp-I, II, and III transfo…
▽ More
Bridge sampling is an effective Monte Carlo method for estimating the ratio of normalizing constants of two probability densities, a routine computational problem in statistics, physics, chemistry, and other fields. The Monte Carlo error of the bridge sampling estimator is determined by the amount of overlap between the two densities. In the case of uni-modal densities, Warp-I, II, and III transformations (Meng and Schilling, 2002) are effective for increasing the initial overlap, but they are less so for multi-modal densities. This paper introduces Warp-U transformations that aim to transform multi-modal densities into Uni-modal ones without altering their normalizing constants. The construction of a Warp-U transformation starts with a Normal (or other convenient) mixture distribution $φ_{\text{mix}}$ that has reasonable overlap with the target density $p$, whose normalizing constant is unknown. The stochastic transformation that maps $φ_{\text{mix}}$ back to its generating distribution $N(0,1)$ is then applied to $p$ yielding its Warp-U version, which we denote $\tilde{p}$. Typically, $\tilde{p}$ is uni-modal and has substantially increased overlap with $N(0,1)$. Furthermore, we prove that the overlap between $\tilde{p}$ and $N(0,1)$ is guaranteed to be no less than the overlap between $p$ and $φ_{\text{mix}}$, in terms of any $f$-divergence. We propose a computationally efficient method to find an appropriate $φ_{\text{mix}}$, and a simple but effective approach to remove the bias which results from estimating the normalizing constants and fitting $φ_{\text{mix}}$ with the same data. We illustrate our findings using 10 and 50 dimensional highly irregular multi-modal densities, and demonstrate how Warp-U sampling can be used to improve the final estimation step of the Generalized Wang-Landau algorithm (Liang, 2005), a powerful sampling and estimation method.
△ Less
Submitted 8 June, 2019; v1 submitted 24 September, 2016;
originally announced September 2016.
-
Bayesian Learning and Predictability in a Stochastic Nonlinear Dynamical Model
Authors:
John Parslow,
Noel Cressie,
Edward P. Campbell,
Emlyn Jones,
Lawrence Murray
Abstract:
Bayesian inference methods are applied within a Bayesian hierarchical modelling framework to the problems of joint state and parameter estimation, and of state forecasting. We explore and demonstrate the ideas in the context of a simple nonlinear marine biogeochemical model. A novel approach is proposed to the formulation of the stochastic process model, in which ecophysiological properties of pla…
▽ More
Bayesian inference methods are applied within a Bayesian hierarchical modelling framework to the problems of joint state and parameter estimation, and of state forecasting. We explore and demonstrate the ideas in the context of a simple nonlinear marine biogeochemical model. A novel approach is proposed to the formulation of the stochastic process model, in which ecophysiological properties of plankton communities are represented by autoregressive stochastic processes. This approach captures the effects of changes in plankton communities over time, and it allows the incorporation of literature metadata on individual species into prior distributions for process model parameters. The approach is applied to a case study at Ocean Station Papa, using Particle Markov chain Monte Carlo computational techniques. The results suggest that, by drawing on objective prior information, it is possible to extract useful information about model state and a subset of parameters, and even to make useful long-term forecasts, based on sparse and noisy observations.
△ Less
Submitted 7 November, 2012;
originally announced November 2012.
-
On Disturbance State-Space Models and the Particle Marginal Metropolis-Hastings Sampler
Authors:
Lawrence M. Murray,
Emlyn M. Jones,
John Parslow
Abstract:
We investigate nonlinear state-space models without a closed-form transition density, and propose reformulating such models over their latent noise variables rather than their latent state variables. In doing so the tractable noise density emerges in place of the intractable transition density. For importance sampling methods such as the auxiliary particle filter, this enables importance weights t…
▽ More
We investigate nonlinear state-space models without a closed-form transition density, and propose reformulating such models over their latent noise variables rather than their latent state variables. In doing so the tractable noise density emerges in place of the intractable transition density. For importance sampling methods such as the auxiliary particle filter, this enables importance weights to be computed where they could not be otherwise. As case studies we take two multivariate marine biogeochemical models and perform state and parameter estimation using the particle marginal Metropolis-Hastings sampler. For the particle filter within this sampler, we compare several proposal strategies over noise variables, all based on lookaheads with the unscented Kalman filter. These strategies are compared using conventional means for assessing Metropolis-Hastings efficiency, as well as with a novel metric called the conditional acceptance rate for assessing the consequences of using an estimated, and not exact, likelihood. Results indicate the utility of reformulating the model over noise variables, particularly for fast-mixing process models.
△ Less
Submitted 10 December, 2013; v1 submitted 28 February, 2012;
originally announced February 2012.
