-
Faster Linear Algebra Algorithms with Structured Random Matrices
Authors:
Chris Camaño,
Ethan N. Epperly,
Raphael A. Meyer,
Joel A. Tropp
Abstract:
To achieve the greatest possible speed, practitioners regularly implement randomized algorithms for low-rank approximation and least-squares regression with structured dimension reduction maps. Despite significant research effort, basic questions remain about the design and analysis of randomized linear algebra algorithms that employ structured random matrices.
This paper develops a new perspect…
▽ More
To achieve the greatest possible speed, practitioners regularly implement randomized algorithms for low-rank approximation and least-squares regression with structured dimension reduction maps. Despite significant research effort, basic questions remain about the design and analysis of randomized linear algebra algorithms that employ structured random matrices.
This paper develops a new perspective on structured dimension reduction, based on the oblivious subspace injection (OSI) property. The OSI property is a relatively weak assumption on a random matrix that holds when the matrix preserves the length of vectors on average and, with high probability, does not annihilate any vector in a low-dimensional subspace. With the OSI abstraction, the analysis of a randomized linear algebra algorithm factors into two parts: (i) proving that the algorithm works when implemented with an OSI; and (ii) proving that a given random matrix model has the OSI property.
This paper develops both parts of the program. First, it analyzes standard randomized algorithms for low-rank approximation and least-squares regression under the OSI assumption. Second, it identifies many examples of OSIs, including random sparse matrices, randomized trigonometric transforms, and random matrices with tensor product structure. These theoretical results imply faster, near-optimal runtimes for several fundamental linear algebra tasks. The paper also provides guidance on implementation, along with empirical evidence that structured random matrices offer exemplary performance for a range of synthetic problems and contemporary scientific applications.
△ Less
Submitted 28 August, 2025;
originally announced August 2025.
-
Debiasing Polynomial and Fourier Regression
Authors:
Chris Camaño,
Raphael A. Meyer,
Kevin Shu
Abstract:
We study the problem of approximating an unknown function $f:\mathbb{R}\to\mathbb{R}$ by a degree-$d$ polynomial using as few function evaluations as possible, where error is measured with respect to a probability distribution $μ$. Existing randomized algorithms achieve near-optimal sample complexities to recover a $ (1+\varepsilon) $-optimal polynomial but produce biased estimates of the best pol…
▽ More
We study the problem of approximating an unknown function $f:\mathbb{R}\to\mathbb{R}$ by a degree-$d$ polynomial using as few function evaluations as possible, where error is measured with respect to a probability distribution $μ$. Existing randomized algorithms achieve near-optimal sample complexities to recover a $ (1+\varepsilon) $-optimal polynomial but produce biased estimates of the best polynomial approximation, which is undesirable.
We propose a simple debiasing method based on a connection between polynomial regression and random matrix theory. Our method involves evaluating $f(λ_1),\ldots,f(λ_{d+1})$ where $λ_1,\ldots,λ_{d+1}$ are the eigenvalues of a suitably designed random complex matrix tailored to the distribution $μ$. Our estimator is unbiased, has near-optimal sample complexity, and experimentally outperforms iid leverage score sampling.
Additionally, our techniques enable us to debias existing methods for approximating a periodic function with a truncated Fourier series with near-optimal sample complexity.
△ Less
Submitted 7 August, 2025;
originally announced August 2025.
-
High-Dimensional Gaussian Process Regression with Soft Kernel Interpolation
Authors:
Chris Camaño,
Daniel Huang
Abstract:
We introduce Soft Kernel Interpolation (SoftKI), a method that combines aspects of Structured Kernel Interpolation (SKI) and variational inducing point methods, to achieve scalable Gaussian Process (GP) regression on high-dimensional datasets. SoftKI approximates a kernel via softmax interpolation from a smaller number of interpolation points learned by optimizing a combination of the SoftKI margi…
▽ More
We introduce Soft Kernel Interpolation (SoftKI), a method that combines aspects of Structured Kernel Interpolation (SKI) and variational inducing point methods, to achieve scalable Gaussian Process (GP) regression on high-dimensional datasets. SoftKI approximates a kernel via softmax interpolation from a smaller number of interpolation points learned by optimizing a combination of the SoftKI marginal log-likelihood (MLL), and when needed, an approximate MLL for improved numerical stability. Consequently, it can overcome the dimensionality scaling challenges that SKI faces when interpolating from a dense and static lattice while retaining the flexibility of variational methods to adapt inducing points to the dataset. We demonstrate the effectiveness of SoftKI across various examples and show that it is competitive with other approximated GP methods when the data dimensionality is modest (around 10).
△ Less
Submitted 16 September, 2025; v1 submitted 28 October, 2024;
originally announced October 2024.
-
Push: Concurrent Probabilistic Programming for Bayesian Deep Learning
Authors:
Daniel Huang,
Chris Camaño,
Jonathan Tsegaye,
Jonathan Austin Gale
Abstract:
We introduce a library called Push that takes a probabilistic programming approach to Bayesian deep learning (BDL). This library enables concurrent execution of BDL inference algorithms on multi-GPU hardware for neural network (NN) models. To accomplish this, Push introduces an abstraction that represents an input NN as a particle. Push enables easy creation of particles so that an input NN can be…
▽ More
We introduce a library called Push that takes a probabilistic programming approach to Bayesian deep learning (BDL). This library enables concurrent execution of BDL inference algorithms on multi-GPU hardware for neural network (NN) models. To accomplish this, Push introduces an abstraction that represents an input NN as a particle. Push enables easy creation of particles so that an input NN can be replicated and particles can communicate asynchronously so that a variety of parameter updates can be expressed, including common BDL algorithms. Our hope is that Push lowers the barrier to experimenting with BDL by streamlining the scaling of particles across GPUs. We evaluate the scaling behavior of particles on single-node multi-GPU devices on vision and scientific machine learning (SciML) tasks.
△ Less
Submitted 29 September, 2023; v1 submitted 10 June, 2023;
originally announced June 2023.
