-
Uniform approximation of common Gaussian process kernels using equispaced Fourier grids
Authors:
Alex Barnett,
Philip Greengard,
Manas Rachh
Abstract:
The high efficiency of a recently proposed method for computing with Gaussian processes relies on expanding a (translationally invariant) covariance kernel into complex exponentials, with frequencies lying on a Cartesian equispaced grid. Here we provide rigorous error bounds for this approximation for two popular kernels -- Matérn and squared exponential -- in terms of the grid spacing and size. T…
▽ More
The high efficiency of a recently proposed method for computing with Gaussian processes relies on expanding a (translationally invariant) covariance kernel into complex exponentials, with frequencies lying on a Cartesian equispaced grid. Here we provide rigorous error bounds for this approximation for two popular kernels -- Matérn and squared exponential -- in terms of the grid spacing and size. The kernel error bounds are uniform over a hypercube centered at the origin. Our tools include a split into aliasing and truncation errors, and bounds on sums of Gaussians or modified Bessel functions over various lattices. For the Matérn case, motivated by numerical study, we conjecture a stronger Frobenius-norm bound on the covariance matrix error for randomly-distributed data points. Lastly, we prove bounds on, and study numerically, the ill-conditioning of the linear systems arising in such regression problems.
△ Less
Submitted 18 May, 2023;
originally announced May 2023.
-
Equispaced Fourier representations for efficient Gaussian process regression from a billion data points
Authors:
Philip Greengard,
Manas Rachh,
Alex Barnett
Abstract:
We introduce a Fourier-based fast algorithm for Gaussian process regression in low dimensions. It approximates a translationally-invariant covariance kernel by complex exponentials on an equispaced Cartesian frequency grid of $M$ nodes. This results in a weight-space $M\times M$ system matrix with Toeplitz structure, which can thus be applied to a vector in ${\mathcal O}(M \log{M})$ operations via…
▽ More
We introduce a Fourier-based fast algorithm for Gaussian process regression in low dimensions. It approximates a translationally-invariant covariance kernel by complex exponentials on an equispaced Cartesian frequency grid of $M$ nodes. This results in a weight-space $M\times M$ system matrix with Toeplitz structure, which can thus be applied to a vector in ${\mathcal O}(M \log{M})$ operations via the fast Fourier transform (FFT), independent of the number of data points $N$. The linear system can be set up in ${\mathcal O}(N + M \log{M})$ operations using nonuniform FFTs. This enables efficient massive-scale regression via an iterative solver, even for kernels with fat-tailed spectral densities (large $M$). We provide bounds on both kernel approximation and posterior mean errors. Numerical experiments for squared-exponential and Matérn kernels in one, two and three dimensions often show 1-2 orders of magnitude acceleration over state-of-the-art rank-structured solvers at comparable accuracy. Our method allows 2D Matérn-$\mbox{$\frac{3}{2}$}$ regression from $N=10^9$ data points to be performed in 2 minutes on a standard desktop, with posterior mean accuracy $10^{-3}$. This opens up spatial statistics applications 100 times larger than previously possible.
△ Less
Submitted 18 May, 2023; v1 submitted 18 October, 2022;
originally announced October 2022.
-
On a linearization of quadratic Wasserstein distance
Authors:
Philip Greengard,
Jeremy G. Hoskins,
Nicholas F. Marshall,
Amit Singer
Abstract:
This paper studies the problem of computing a linear approximation of quadratic Wasserstein distance $W_2$. In particular, we compute an approximation of the negative homogeneous weighted Sobolev norm whose connection to Wasserstein distance follows from a classic linearization of a general Monge-Ampére equation. Our contribution is threefold. First, we provide expository material on this classic…
▽ More
This paper studies the problem of computing a linear approximation of quadratic Wasserstein distance $W_2$. In particular, we compute an approximation of the negative homogeneous weighted Sobolev norm whose connection to Wasserstein distance follows from a classic linearization of a general Monge-Ampére equation. Our contribution is threefold. First, we provide expository material on this classic linearization of Wasserstein distance including a quantitative error estimate. Second, we reduce the computational problem to solving an elliptic boundary value problem involving the Witten Laplacian, which is a Schrödinger operator of the form $H = -Δ+ V$, and describe an associated embedding. Third, for the case of probability distributions on the unit square $[0,1]^2$ represented by $n \times n$ arrays we present a fast code demonstrating our approach. Several numerical examples are presented.
△ Less
Submitted 1 March, 2022; v1 submitted 31 January, 2022;
originally announced January 2022.
-
Efficient Fourier representations of families of Gaussian processes
Authors:
Philip Greengard
Abstract:
We introduce a class of algorithms for constructing Fourier representations of Gaussian processes in $1$ dimension that are valid over ranges of hyperparameter values. The scaling and frequencies of the Fourier basis functions are evaluated numerically via generalized quadratures. The representations introduced allow for $O(m^3)$ inference, independent of $N$, for all hyperparameters in the user-s…
▽ More
We introduce a class of algorithms for constructing Fourier representations of Gaussian processes in $1$ dimension that are valid over ranges of hyperparameter values. The scaling and frequencies of the Fourier basis functions are evaluated numerically via generalized quadratures. The representations introduced allow for $O(m^3)$ inference, independent of $N$, for all hyperparameters in the user-specified range after $O(N + m^2\log{m})$ precomputation where $N$, the number of data points, is usually significantly larger than $m$, the number of basis functions. Inference independent of $N$ for various hyperparameters is facilitated by generalized quadratures, and the $O(N + m^2\log{m})$ precomputation is achieved with the non-uniform FFT. Numerical results are provided for Matérn kernels with $ν\in [3/2, 7/2]$ and lengthscale $ρ\in [0.1, 0.5]$ and squared-exponential kernels with lengthscale $ρ\in [0.1, 0.5]$. The algorithms of this paper generalize mathematically to higher dimensions, though they suffer from the standard curse of dimensionality.
△ Less
Submitted 3 June, 2024; v1 submitted 28 September, 2021;
originally announced September 2021.
-
Efficient reduced-rank methods for Gaussian processes with eigenfunction expansions
Authors:
Philip Greengard,
Michael O'Neil
Abstract:
In this work we introduce a reduced-rank algorithm for Gaussian process regression. Our numerical scheme converts a Gaussian process on a user-specified interval to its Karhunen-Loève expansion, the $L^2$-optimal reduced-rank representation. Numerical evaluation of the Karhunen-Loève expansion is performed once during precomputation and involves computing a numerical eigendecomposition of an integ…
▽ More
In this work we introduce a reduced-rank algorithm for Gaussian process regression. Our numerical scheme converts a Gaussian process on a user-specified interval to its Karhunen-Loève expansion, the $L^2$-optimal reduced-rank representation. Numerical evaluation of the Karhunen-Loève expansion is performed once during precomputation and involves computing a numerical eigendecomposition of an integral operator whose kernel is the covariance function of the Gaussian process. The Karhunen-Loève expansion is independent of observed data and depends only on the covariance kernel and the size of the interval on which the Gaussian process is defined. The scheme of this paper does not require translation invariance of the covariance kernel. We also introduce a class of fast algorithms for Bayesian fitting of hyperparameters, and demonstrate the performance of our algorithms with numerical experiments in one and two dimensions. Extensions to higher dimensions are mathematically straightforward but suffer from the standard curses of high dimensions.
△ Less
Submitted 24 August, 2022; v1 submitted 12 August, 2021;
originally announced August 2021.
-
The piranha problem: Large effects swimming in a small pond
Authors:
Christopher Tosh,
Philip Greengard,
Ben Goodrich,
Andrew Gelman,
Aki Vehtari,
Daniel Hsu
Abstract:
In some scientific fields, it is common to have certain variables of interest that are of particular importance and for which there are many studies indicating a relationship with different explanatory variables. In such cases, particularly those where no relationships are known among the explanatory variables, it is worth asking under what conditions it is possible for all such claimed effects to…
▽ More
In some scientific fields, it is common to have certain variables of interest that are of particular importance and for which there are many studies indicating a relationship with different explanatory variables. In such cases, particularly those where no relationships are known among the explanatory variables, it is worth asking under what conditions it is possible for all such claimed effects to exist simultaneously. This paper addresses this question by reviewing some theorems from multivariate analysis showing that, unless the explanatory variables also have sizable dependencies with each other, it is impossible to have many such large effects. We discuss implications for the replication crisis in social science.
△ Less
Submitted 23 July, 2024; v1 submitted 27 May, 2021;
originally announced May 2021.
-
A Fast Linear Regression via SVD and Marginalization
Authors:
Philip Greengard,
Andrew Gelman,
Aki Vehtari
Abstract:
We describe a numerical scheme for evaluating the posterior moments of Bayesian linear regression models with partial pooling of the coefficients. The principal analytical tool of the evaluation is a change of basis from coefficient space to the space of singular vectors of the matrix of predictors. After this change of basis and an analytical integration, we reduce the problem of finding moments…
▽ More
We describe a numerical scheme for evaluating the posterior moments of Bayesian linear regression models with partial pooling of the coefficients. The principal analytical tool of the evaluation is a change of basis from coefficient space to the space of singular vectors of the matrix of predictors. After this change of basis and an analytical integration, we reduce the problem of finding moments of a density over k + m dimensions, to finding moments of an m-dimensional density, where k is the number of coefficients and k + m is the dimension of the posterior. Moments can then be computed using, for example, MCMC, the trapezoid rule, or adaptive Gaussian quadrature. An evaluation of the SVD of the matrix of predictors is the dominant computational cost and is performed once during the precomputation stage. We demonstrate numerical results of the algorithm. The scheme described in this paper generalizes naturally to multilevel and multi-group hierarchical regression models where normal-normal parameters appear.
△ Less
Submitted 9 November, 2020;
originally announced November 2020.
-
Generalized prolate spheroidal functions: algorithms and analysis
Authors:
Philip Greengard
Abstract:
Generalized prolate spheroidal functions (GPSFs) arise naturally in the study of bandlimited functions as the eigenfunctions of a certain truncated Fourier transform. In one dimension, the theory of GPSFs (typically referred to as prolate spheroidal wave functions) has a long history and is fairly complete. Furthermore, more recent work has led to the development of numerical algorithms for their…
▽ More
Generalized prolate spheroidal functions (GPSFs) arise naturally in the study of bandlimited functions as the eigenfunctions of a certain truncated Fourier transform. In one dimension, the theory of GPSFs (typically referred to as prolate spheroidal wave functions) has a long history and is fairly complete. Furthermore, more recent work has led to the development of numerical algorithms for their computation and use in applications. In this paper we consider the more general problem, extending the one dimensional analysis and algorithms to the case of arbitrary dimension. Specifically, we introduce algorithms for efficient evaluation of GPSFs and their corresponding eigenvalues, quadrature rules for bandlimited functions, formulae for interpolation via GPSF expansion, and various analytical properties of GPSFs. We illustrate the numerical and analytical results with several numerical examples.
△ Less
Submitted 6 May, 2024; v1 submitted 6 November, 2018;
originally announced November 2018.
-
Zernike Polynomials: Evaluation, Quadrature, and Interpolation
Authors:
Philip Greengard,
Kirill Serkh
Abstract:
Zernike polynomials are a basis of orthogonal polynomials on the unit disk that are a natural basis for representing smooth functions. They arise in a number of applications including optics and atmospheric sciences. In this paper, we provide a self-contained reference on Zernike polynomials, algorithms for evaluating them, and what appear to be new numerical schemes for quadrature and interpolati…
▽ More
Zernike polynomials are a basis of orthogonal polynomials on the unit disk that are a natural basis for representing smooth functions. They arise in a number of applications including optics and atmospheric sciences. In this paper, we provide a self-contained reference on Zernike polynomials, algorithms for evaluating them, and what appear to be new numerical schemes for quadrature and interpolation. We also introduce new properties of Zernike polynomials in higher dimensions. The quadrature rule and interpolation scheme use a tensor product of equispaced nodes in the angular direction and roots of certain Jacobi polynomials in the radial direction. An algorithm for finding the roots of these Jacobi polynomials is also described. The performance of the interpolation and quadrature schemes is illustrated through numerical experiments. Discussions of higher dimensional Zernike polynomials are included in appendices.
△ Less
Submitted 6 November, 2018;
originally announced November 2018.
