-
Extremal bounds for Gaussian trace estimation
Authors:
Eric Hallman
Abstract:
This work derives extremal tail bounds for the Gaussian trace estimator applied to a real symmetric matrix. We define a partial ordering on the eigenvalues, so that when a matrix has greater spectrum under this ordering, its estimator will have worse tail bounds. This is done for two families of matrices: positive semidefinite matrices with bounded effective rank, and indefinite matrices with boun…
▽ More
This work derives extremal tail bounds for the Gaussian trace estimator applied to a real symmetric matrix. We define a partial ordering on the eigenvalues, so that when a matrix has greater spectrum under this ordering, its estimator will have worse tail bounds. This is done for two families of matrices: positive semidefinite matrices with bounded effective rank, and indefinite matrices with bounded 2-norm and fixed Frobenius norm. In each case, the tail region is defined rigorously and is constant for a given family.
△ Less
Submitted 23 November, 2024;
originally announced November 2024.
-
Krylov-aware stochastic trace estimation
Authors:
Tyler Chen,
Eric Hallman
Abstract:
We introduce an algorithm for estimating the trace of a matrix function $f(\mathbf{A})$ using implicit products with a symmetric matrix $\mathbf{A}$. Existing methods for implicit trace estimation of a matrix function tend to treat matrix-vector products with $f(\mathbf{A})$ as a black-box to be computed by a Krylov subspace method. Like other recent algorithms for implicit trace estimation, our a…
▽ More
We introduce an algorithm for estimating the trace of a matrix function $f(\mathbf{A})$ using implicit products with a symmetric matrix $\mathbf{A}$. Existing methods for implicit trace estimation of a matrix function tend to treat matrix-vector products with $f(\mathbf{A})$ as a black-box to be computed by a Krylov subspace method. Like other recent algorithms for implicit trace estimation, our approach is based on a combination of deflation and stochastic trace estimation. However, we take a closer look at how products with $f(\mathbf{A})$ are integrated into these approaches which enables several efficiencies not present in previously studied methods. In particular, we describe a Krylov subspace method for computing a low-rank approximation of a matrix function by a computationally efficient projection onto Krylov subspace.
△ Less
Submitted 28 August, 2023; v1 submitted 3 May, 2022;
originally announced May 2022.
-
Precision-aware Deterministic and Probabilistic Error Bounds for Floating Point Summation
Authors:
Eric Hallman,
Ilse C. F. Ipsen
Abstract:
We analyze the forward error in the floating point summation of real numbers, for computations in low precision or extreme-scale problem dimensions that push the limits of the precision. We present a systematic recurrence for a martingale on a computational tree, which leads to explicit and interpretable bounds without asymptotic big-O terms. Two probability parameters strengthen the precision-awa…
▽ More
We analyze the forward error in the floating point summation of real numbers, for computations in low precision or extreme-scale problem dimensions that push the limits of the precision. We present a systematic recurrence for a martingale on a computational tree, which leads to explicit and interpretable bounds without asymptotic big-O terms. Two probability parameters strengthen the precision-awareness of our bounds: one parameter controls the first order terms in the summation error, while the second one is designed for controlling higher order terms in low precision or extreme-scale problem dimensions. Our systematic approach yields new deterministic and probabilistic error bounds for three classes of mono-precision algorithms: general summation, shifted general summation, and compensated (sequential) summation. Extension of our systematic error analysis to mixed-precision summation algorithms that allow any number of precisions yields the first probabilistic bounds for the mixed-precision FABsum algorithm. Numerical experiments illustrate that the probabilistic bounds are accurate, and that among the three classes of mono-precision algorithms, compensated summation is generally the most accurate. As for mixed precision algorithms, our recommendation is to minimize the magnitude of intermediate partial sums relative to the precision in which they are computed.
△ Less
Submitted 29 March, 2022;
originally announced March 2022.
-
Monte Carlo Methods for Estimating the Diagonal of a Real Symmetric Matrix
Authors:
Eric Hallman,
Ilse C. F. Ipsen,
Arvind Saibaba
Abstract:
For real symmetric matrices that are accessible only through matrix vector products, we present Monte Carlo estimators for computing the diagonal elements. Our probabilistic bounds for normwise absolute and relative errors apply to Monte Carlo estimators based on random Rademacher, sparse Rademacher, normalized and unnormalized Gaussian vectors, and to vectors with bounded fourth moments. The nove…
▽ More
For real symmetric matrices that are accessible only through matrix vector products, we present Monte Carlo estimators for computing the diagonal elements. Our probabilistic bounds for normwise absolute and relative errors apply to Monte Carlo estimators based on random Rademacher, sparse Rademacher, normalized and unnormalized Gaussian vectors, and to vectors with bounded fourth moments. The novel use of matrix concentration inequalities in our proofs represents a systematic model for future analyses. Our bounds mostly do not depend on the matrix dimension, target different error measures than existing work, and imply that the accuracy of the estimators increases with the diagonal dominance of the matrix. An application to derivative-based global sensitivity metrics corroborates this, as do numerical experiments on synthetic test matrices. We recommend against the use in practice of sparse Rademacher vectors, which are the basis for many randomized sketching and sampling algorithms, because they tend to deliver barely a digit of accuracy even under large sampling amounts.
△ Less
Submitted 17 March, 2022; v1 submitted 6 February, 2022;
originally announced February 2022.
-
Randomized algorithms for rounding in the Tensor-Train format
Authors:
Hussam Al Daas,
Grey Ballard,
Paul Cazeaux,
Eric Hallman,
Agnieszka Miedlar,
Mirjeta Pasha,
Tim W. Reid,
Arvind K. Saibaba
Abstract:
The Tensor-Train (TT) format is a highly compact low-rank representation for high-dimensional tensors. TT is particularly useful when representing approximations to the solutions of certain types of parametrized partial differential equations. For many of these problems, computing the solution explicitly would require an infeasible amount of memory and computational time. While the TT format makes…
▽ More
The Tensor-Train (TT) format is a highly compact low-rank representation for high-dimensional tensors. TT is particularly useful when representing approximations to the solutions of certain types of parametrized partial differential equations. For many of these problems, computing the solution explicitly would require an infeasible amount of memory and computational time. While the TT format makes these problems tractable, iterative techniques for solving the PDEs must be adapted to perform arithmetic while maintaining the implicit structure. The fundamental operation used to maintain feasible memory and computational time is called rounding, which truncates the internal ranks of a tensor already in TT format. We propose several randomized algorithms for this task that are generalizations of randomized low-rank matrix approximation algorithms and provide significant reduction in computation compared to deterministic TT-rounding algorithms. Randomization is particularly effective in the case of rounding a sum of TT-tensors (where we observe 20x speedup), which is the bottleneck computation in the adaptation of GMRES to vectors in TT format. We present the randomized algorithms and compare their empirical accuracy and computational time with deterministic alternatives.
△ Less
Submitted 8 October, 2021;
originally announced October 2021.
-
Deterministic and Probabilistic Error Bounds for Floating Point Summation Algorithms
Authors:
Eric Hallman,
Ilse C. F. Ipsen
Abstract:
We analyse the forward error in the floating point summation of real numbers, from algorithms that do not require recourse to higher precision or better hardware. We derive informative explicit expressions, and new deterministic and probabilistic bounds for errors in three classes of algorithms: general summation,shifted general summation, and compensated (sequential) summation. Our probabilistic…
▽ More
We analyse the forward error in the floating point summation of real numbers, from algorithms that do not require recourse to higher precision or better hardware. We derive informative explicit expressions, and new deterministic and probabilistic bounds for errors in three classes of algorithms: general summation,shifted general summation, and compensated (sequential) summation. Our probabilistic bounds for general and shifted general summation hold to all orders. For compensated summation, we also present deterministic and probabilistic first and second order bounds, with a first order bound that differs from existing ones. Numerical experiments illustrate that the bounds are informative and that among the three algorithm classes, compensated summation is generally the most accurate method.
△ Less
Submitted 4 July, 2021;
originally announced July 2021.
-
A Refined Probabilistic Error Bound for Sums
Authors:
Eric Hallman
Abstract:
This paper considers a probabilistic model for floating-point computation in which the roundoff errors are represented by bounded random variables with mean zero. Using this model, a probabilistic bound is derived for the forward error of the computed sum of n real numbers. This work improves upon existing probabilistic bounds by holding to all orders, and as a result provides informative bounds f…
▽ More
This paper considers a probabilistic model for floating-point computation in which the roundoff errors are represented by bounded random variables with mean zero. Using this model, a probabilistic bound is derived for the forward error of the computed sum of n real numbers. This work improves upon existing probabilistic bounds by holding to all orders, and as a result provides informative bounds for larger problem sizes.
△ Less
Submitted 13 April, 2021;
originally announced April 2021.
-
A Multilevel Approach to Stochastic Trace Estimation
Authors:
Eric Hallman,
Devon Troester
Abstract:
This article presents a randomized matrix-free method for approximating the trace of $f({\bf A})$, where ${\bf A}$ is a large symmetric matrix and $f$ is a function analytic in a closed interval containing the eigenvalues of ${\bf A}$. Our method uses a combination of stochastic trace estimation (i.e., Hutchinson's method), Chebyshev approximation, and multilevel Monte Carlo techniques. We establi…
▽ More
This article presents a randomized matrix-free method for approximating the trace of $f({\bf A})$, where ${\bf A}$ is a large symmetric matrix and $f$ is a function analytic in a closed interval containing the eigenvalues of ${\bf A}$. Our method uses a combination of stochastic trace estimation (i.e., Hutchinson's method), Chebyshev approximation, and multilevel Monte Carlo techniques. We establish general bounds on the approximation error of this method by extending an existing error bound for Hutchinson's method to multilevel trace estimators. Numerical experiments are conducted for common applications such as estimating the log-determinant, nuclear norm, and Estrada index, and triangle counting in graphs. We find that using multilevel techniques can substantially reduce the variance of existing single-level estimators.
△ Less
Submitted 18 March, 2021;
originally announced March 2021.
-
A Block Bidiagonalization Method for Fixed-Accuracy Low-Rank Matrix Approximation
Authors:
Eric Hallman
Abstract:
We present randUBV, a randomized algorithm for matrix sketching based on the block Lanzcos bidiagonalization process. Given a matrix $\bf{A}$, it produces a low-rank approximation of the form ${\bf UBV}^T$, where $\bf{U}$ and $\bf{V}$ have orthonormal columns in exact arithmetic and $\bf{B}$ is block bidiagonal. In finite precision, the columns of both ${\bf U}$ and ${\bf V}$ will be close to orth…
▽ More
We present randUBV, a randomized algorithm for matrix sketching based on the block Lanzcos bidiagonalization process. Given a matrix $\bf{A}$, it produces a low-rank approximation of the form ${\bf UBV}^T$, where $\bf{U}$ and $\bf{V}$ have orthonormal columns in exact arithmetic and $\bf{B}$ is block bidiagonal. In finite precision, the columns of both ${\bf U}$ and ${\bf V}$ will be close to orthonormal. Our algorithm is closely related to the randQB algorithms of Yu, Gu, and Li (2018) in that the entries of $\bf{B}$ are incrementally generated and the Frobenius norm approximation error may be efficiently estimated. Our algorithm is therefore suitable for the fixed-accuracy problem, and so is designed to terminate as soon as a user input error tolerance is reached. Numerical experiments suggest that the block Lanczos method is generally competitive with or superior to algorithms that use power iteration, even when $\bf{A}$ has significant clusters of singular values.
△ Less
Submitted 8 February, 2021; v1 submitted 4 January, 2021;
originally announced January 2021.
-
Faster Stochastic Trace Estimation with a Chebyshev Product Identity
Authors:
Eric Hallman
Abstract:
Methods for stochastic trace estimation often require the repeated evaluation of expressions of the form $z^T p_n(A)z$, where $A$ is a symmetric matrix and $p_n$ is a degree $n$ polynomial written in the standard or Chebyshev basis. We show how to evaluate these expressions using only $\lceil n/2\rceil$ matrix-vector products, thus substantially reducing the cost of existing trace estimation algor…
▽ More
Methods for stochastic trace estimation often require the repeated evaluation of expressions of the form $z^T p_n(A)z$, where $A$ is a symmetric matrix and $p_n$ is a degree $n$ polynomial written in the standard or Chebyshev basis. We show how to evaluate these expressions using only $\lceil n/2\rceil$ matrix-vector products, thus substantially reducing the cost of existing trace estimation algorithms that use Chebyshev interpolation or Taylor series.
△ Less
Submitted 1 January, 2021;
originally announced January 2021.
