-
Stein's Method Meets Computational Statistics: A Review of Some Recent Developments
Authors:
Andreas Anastasiou,
Alessandro Barp,
François-Xavier Briol,
Bruno Ebner,
Robert E. Gaunt,
Fatemeh Ghaderinezhad,
Jackson Gorham,
Arthur Gretton,
Christophe Ley,
Qiang Liu,
Lester Mackey,
Chris. J. Oates,
Gesine Reinert,
Yvik Swan
Abstract:
Stein's method compares probability distributions through the study of a class of linear operators called Stein operators. While mainly studied in probability and used to underpin theoretical statistics, Stein's method has led to significant advances in computational statistics in recent years. The goal of this survey is to bring together some of these recent developments and, in doing so, to stim…
▽ More
Stein's method compares probability distributions through the study of a class of linear operators called Stein operators. While mainly studied in probability and used to underpin theoretical statistics, Stein's method has led to significant advances in computational statistics in recent years. The goal of this survey is to bring together some of these recent developments and, in doing so, to stimulate further research into the successful field of Stein's method and statistics. The topics we discuss include tools to benchmark and compare sampling methods such as approximate Markov chain Monte Carlo, deterministic alternatives to sampling methods, control variate techniques, parameter estimation and goodness-of-fit testing.
△ Less
Submitted 22 June, 2022; v1 submitted 7 May, 2021;
originally announced May 2021.
-
Classifying toric 3-fold codes of dimensions 4 and 5
Authors:
Tori Braun,
James Carzon,
Jenna Gorham,
Kelly Jabbusch
Abstract:
A toric code is an error-correcting code determined by a toric variety or its associated integral convex polytope. We investigate $4$- and $5$-dimensional toric $3$-fold codes, which are codes arising from polytopes in $\mathbf{R}^3$ with four and five lattice points, respectively. By computing the minimum distances of each code, we fully classify the $4$-dimensional codes. We further present prog…
▽ More
A toric code is an error-correcting code determined by a toric variety or its associated integral convex polytope. We investigate $4$- and $5$-dimensional toric $3$-fold codes, which are codes arising from polytopes in $\mathbf{R}^3$ with four and five lattice points, respectively. By computing the minimum distances of each code, we fully classify the $4$-dimensional codes. We further present progress toward the same goal for dimension $5$ codes. In particular, we classify the $5$-dimensional toric $3$-fold codes arising from polytopes of width 1.
△ Less
Submitted 30 March, 2021;
originally announced March 2021.
-
Stochastic Stein Discrepancies
Authors:
Jackson Gorham,
Anant Raj,
Lester Mackey
Abstract:
Stein discrepancies (SDs) monitor convergence and non-convergence in approximate inference when exact integration and sampling are intractable. However, the computation of a Stein discrepancy can be prohibitive if the Stein operator - often a sum over likelihood terms or potentials - is expensive to evaluate. To address this deficiency, we show that stochastic Stein discrepancies (SSDs) based on s…
▽ More
Stein discrepancies (SDs) monitor convergence and non-convergence in approximate inference when exact integration and sampling are intractable. However, the computation of a Stein discrepancy can be prohibitive if the Stein operator - often a sum over likelihood terms or potentials - is expensive to evaluate. To address this deficiency, we show that stochastic Stein discrepancies (SSDs) based on subsampled approximations of the Stein operator inherit the convergence control properties of standard SDs with probability 1. Along the way, we establish the convergence of Stein variational gradient descent (SVGD) on unbounded domains, resolving an open question of Liu (2017). In our experiments with biased Markov chain Monte Carlo (MCMC) hyperparameter tuning, approximate MCMC sampler selection, and stochastic SVGD, SSDs deliver comparable inferences to standard SDs with orders of magnitude fewer likelihood evaluations.
△ Less
Submitted 22 October, 2020; v1 submitted 6 July, 2020;
originally announced July 2020.
-
Stein Point Markov Chain Monte Carlo
Authors:
Wilson Ye Chen,
Alessandro Barp,
François-Xavier Briol,
Jackson Gorham,
Mark Girolami,
Lester Mackey,
Chris. J. Oates
Abstract:
An important task in machine learning and statistics is the approximation of a probability measure by an empirical measure supported on a discrete point set. Stein Points are a class of algorithms for this task, which proceed by sequentially minimising a Stein discrepancy between the empirical measure and the target and, hence, require the solution of a non-convex optimisation problem to obtain ea…
▽ More
An important task in machine learning and statistics is the approximation of a probability measure by an empirical measure supported on a discrete point set. Stein Points are a class of algorithms for this task, which proceed by sequentially minimising a Stein discrepancy between the empirical measure and the target and, hence, require the solution of a non-convex optimisation problem to obtain each new point. This paper removes the need to solve this optimisation problem by, instead, selecting each new point based on a Markov chain sample path. This significantly reduces the computational cost of Stein Points and leads to a suite of algorithms that are straightforward to implement. The new algorithms are illustrated on a set of challenging Bayesian inference problems, and rigorous theoretical guarantees of consistency are established.
△ Less
Submitted 14 September, 2020; v1 submitted 9 May, 2019;
originally announced May 2019.
-
Measuring Sample Quality with Diffusions
Authors:
Jackson Gorham,
Andrew B. Duncan,
Sebastian J. Vollmer,
Lester Mackey
Abstract:
Stein's method for measuring convergence to a continuous target distribution relies on an operator characterizing the target and Stein factor bounds on the solutions of an associated differential equation. While such operators and bounds are readily available for a diversity of univariate targets, few multivariate targets have been analyzed. We introduce a new class of characterizing operators bas…
▽ More
Stein's method for measuring convergence to a continuous target distribution relies on an operator characterizing the target and Stein factor bounds on the solutions of an associated differential equation. While such operators and bounds are readily available for a diversity of univariate targets, few multivariate targets have been analyzed. We introduce a new class of characterizing operators based on Ito diffusions and develop explicit multivariate Stein factor bounds for any target with a fast-coupling Ito diffusion. As example applications, we develop computable and convergence-determining diffusion Stein discrepancies for log-concave, heavy-tailed, and multimodal targets and use these quality measures to select the hyperparameters of biased Markov chain Monte Carlo (MCMC) samplers, compare random and deterministic quadrature rules, and quantify bias-variance tradeoffs in approximate MCMC. Our results establish a near-linear relationship between diffusion Stein discrepancies and Wasserstein distances, improving upon past work even for strongly log-concave targets. The exposed relationship between Stein factors and Markov process coupling may be of independent interest.
△ Less
Submitted 12 November, 2018; v1 submitted 21 November, 2016;
originally announced November 2016.
-
Multivariate Stein Factors for a Class of Strongly Log-concave Distributions
Authors:
Lester Mackey,
Jackson Gorham
Abstract:
We establish uniform bounds on the low-order derivatives of Stein equation solutions for a broad class of multivariate, strongly log-concave target distributions. These "Stein factor" bounds deliver control over Wasserstein and related smooth function distances and are well-suited to analyzing the computable Stein discrepancy measures of Gorham and Mackey. Our arguments of proof are probabilistic…
▽ More
We establish uniform bounds on the low-order derivatives of Stein equation solutions for a broad class of multivariate, strongly log-concave target distributions. These "Stein factor" bounds deliver control over Wasserstein and related smooth function distances and are well-suited to analyzing the computable Stein discrepancy measures of Gorham and Mackey. Our arguments of proof are probabilistic and feature the synchronous coupling of multiple overdamped Langevin diffusions.
△ Less
Submitted 22 November, 2016; v1 submitted 23 December, 2015;
originally announced December 2015.
-
Measuring Sample Quality with Stein's Method
Authors:
Jackson Gorham,
Lester Mackey
Abstract:
To improve the efficiency of Monte Carlo estimation, practitioners are turning to biased Markov chain Monte Carlo procedures that trade off asymptotic exactness for computational speed. The reasoning is sound: a reduction in variance due to more rapid sampling can outweigh the bias introduced. However, the inexactness creates new challenges for sampler and parameter selection, since standard measu…
▽ More
To improve the efficiency of Monte Carlo estimation, practitioners are turning to biased Markov chain Monte Carlo procedures that trade off asymptotic exactness for computational speed. The reasoning is sound: a reduction in variance due to more rapid sampling can outweigh the bias introduced. However, the inexactness creates new challenges for sampler and parameter selection, since standard measures of sample quality like effective sample size do not account for asymptotic bias. To address these challenges, we introduce a new computable quality measure based on Stein's method that quantifies the maximum discrepancy between sample and target expectations over a large class of test functions. We use our tool to compare exact, biased, and deterministic sample sequences and illustrate applications to hyperparameter selection, convergence rate assessment, and quantifying bias-variance tradeoffs in posterior inference.
△ Less
Submitted 31 December, 2018; v1 submitted 9 June, 2015;
originally announced June 2015.
-
Computational topology for configuration spaces of hard disks
Authors:
Gunnar Carlsson,
Jackson Gorham,
Matthew Kahle,
Jeremy Mason
Abstract:
We explore the topology of configuration spaces of hard disks experimentally, and show that several changes in the topology can already be observed with a small number of particles. The results illustrate a theorem of Baryshnikov, Bubenik, and Kahle that critical points correspond to configurations of disks with balanced mechanical stresses, and suggest conjectures about the asymptotic topology as…
▽ More
We explore the topology of configuration spaces of hard disks experimentally, and show that several changes in the topology can already be observed with a small number of particles. The results illustrate a theorem of Baryshnikov, Bubenik, and Kahle that critical points correspond to configurations of disks with balanced mechanical stresses, and suggest conjectures about the asymptotic topology as the number of disks tends to infinity.
△ Less
Submitted 13 November, 2011; v1 submitted 29 August, 2011;
originally announced August 2011.
