-
Adaptive sparse variational approximations for Gaussian process regression
Authors:
Dennis Nieman,
Botond Szabó
Abstract:
Accurate tuning of hyperparameters is crucial to ensure that models can generalise effectively across different settings. In this paper, we present theoretical guarantees for hyperparameter selection using variational Bayes in the nonparametric regression model. We construct a variational approximation to a hierarchical Bayes procedure, and derive upper bounds for the contraction rate of the varia…
▽ More
Accurate tuning of hyperparameters is crucial to ensure that models can generalise effectively across different settings. In this paper, we present theoretical guarantees for hyperparameter selection using variational Bayes in the nonparametric regression model. We construct a variational approximation to a hierarchical Bayes procedure, and derive upper bounds for the contraction rate of the variational posterior in an abstract setting. The theory is applied to various Gaussian process priors and variational classes, resulting in minimax optimal rates. Our theoretical results are accompanied with numerical analysis both on synthetic and real world data sets.
△ Less
Submitted 4 April, 2025;
originally announced April 2025.
-
Uncertainty quantification for sparse spectral variational approximations in Gaussian process regression
Authors:
Dennis Nieman,
Botond Szabo,
Harry van Zanten
Abstract:
We investigate the frequentist guarantees of the variational sparse Gaussian process regression model. In the theoretical analysis, we focus on the variational approach with spectral features as inducing variables. We derive guarantees and limitations for the frequentist coverage of the resulting variational credible sets. We also derive sufficient and necessary lower bounds for the number of indu…
▽ More
We investigate the frequentist guarantees of the variational sparse Gaussian process regression model. In the theoretical analysis, we focus on the variational approach with spectral features as inducing variables. We derive guarantees and limitations for the frequentist coverage of the resulting variational credible sets. We also derive sufficient and necessary lower bounds for the number of inducing variables required to achieve minimax posterior contraction rates. The implications of these results are demonstrated for different choices of priors. In a numerical analysis we consider a wider range of inducing variable methods and observe similar phenomena beyond the scope of our theoretical findings.
△ Less
Submitted 28 September, 2023; v1 submitted 21 December, 2022;
originally announced December 2022.
-
Input estimation from discrete workload observations in a Lévy-driven storage system
Authors:
Dennis Nieman,
Michel Mandjes,
Liron Ravner
Abstract:
Our goal is to estimate the characteristic exponent of the input to a Lévy-driven storage system from a sample of equispaced workload observations. The estimator relies on an approximate moment equation associated with the Laplace-Stieltjes transform of the workload at exponentially distributed sampling times. The estimator is pointwise consistent for any observation grid. Moreover, a high frequen…
▽ More
Our goal is to estimate the characteristic exponent of the input to a Lévy-driven storage system from a sample of equispaced workload observations. The estimator relies on an approximate moment equation associated with the Laplace-Stieltjes transform of the workload at exponentially distributed sampling times. The estimator is pointwise consistent for any observation grid. Moreover, a high frequency sampling scheme yields asymptotically normal estimation errors for a class of input processes. A resampling scheme that uses the available information in a more efficient manner is suggested and assessed via simulation experiments.
△ Less
Submitted 28 August, 2024; v1 submitted 20 May, 2022;
originally announced May 2022.
-
Contraction rates for sparse variational approximations in Gaussian process regression
Authors:
Dennis Nieman,
Botond Szabo,
Harry van Zanten
Abstract:
We study the theoretical properties of a variational Bayes method in the Gaussian Process regression model. We consider the inducing variables method introduced by Titsias (2009a) and derive sufficient conditions for obtaining contraction rates for the corresponding variational Bayes (VB) posterior. As examples we show that for three particular covariance kernels (Matérn, squared exponential, rand…
▽ More
We study the theoretical properties of a variational Bayes method in the Gaussian Process regression model. We consider the inducing variables method introduced by Titsias (2009a) and derive sufficient conditions for obtaining contraction rates for the corresponding variational Bayes (VB) posterior. As examples we show that for three particular covariance kernels (Matérn, squared exponential, random series prior) the VB approach can achieve optimal, minimax contraction rates for a sufficiently large number of appropriately chosen inducing variables. The theoretical findings are demonstrated by numerical experiments.
△ Less
Submitted 30 March, 2022; v1 submitted 22 September, 2021;
originally announced September 2021.
