-
Angle-free cluster robust Ritz value bounds for restarted block eigensolvers
Authors:
Ming Zhou,
Andrew V. Knyazev,
Klaus Neymeyr
Abstract:
Convergence rates of block iterations for solving eigenvalue problems typically measure errors of Ritz values approximating eigenvalues. The errors of the Ritz values are commonly bounded in terms of principal angles between the initial or iterative subspace and the invariant subspace associated with the target eigenvalues. Such bounds thus cannot be applied repeatedly as needed for restarted bloc…
▽ More
Convergence rates of block iterations for solving eigenvalue problems typically measure errors of Ritz values approximating eigenvalues. The errors of the Ritz values are commonly bounded in terms of principal angles between the initial or iterative subspace and the invariant subspace associated with the target eigenvalues. Such bounds thus cannot be applied repeatedly as needed for restarted block eigensolvers, since the left- and right-hand sides of the bounds use different terms. They must be combined with additional bounds which could cause an overestimation. Alternative repeatable bounds that are angle-free and depend only on the errors of the Ritz values have been pioneered for Hermitian eigenvalue problems in doi:10.1515/rnam.1987.2.5.371 but only for a single extreme Ritz value. We extend this result to all Ritz values and achieve robustness for clustered eigenvalues by utilizing nonconsecutive eigenvalues. Our new bounds cover the restarted block Lanczos method and its modifications with shift-and-invert and deflation, and are numerically advantageous.
△ Less
Submitted 2 November, 2022;
originally announced November 2022.
-
Convergence analysis of a block preconditioned steepest descent eigensolver with implicit deflation
Authors:
Ming Zhou,
Zhaojun Bai,
Yunfeng Cai,
Klaus Neymeyr
Abstract:
Gradient-type iterative methods for solving Hermitian eigenvalue problems can be accelerated by using preconditioning and deflation techniques. A preconditioned steepest descent iteration with implicit deflation (PSD-id) is one of such methods. The convergence behavior of the PSD-id is recently investigated based on the pioneering work of Samokish on the preconditioned steepest descent method (PSD…
▽ More
Gradient-type iterative methods for solving Hermitian eigenvalue problems can be accelerated by using preconditioning and deflation techniques. A preconditioned steepest descent iteration with implicit deflation (PSD-id) is one of such methods. The convergence behavior of the PSD-id is recently investigated based on the pioneering work of Samokish on the preconditioned steepest descent method (PSD). The resulting non-asymptotic estimates indicate a superlinear convergence of the PSD-id under strong assumptions on the initial guess. The present paper utilizes an alternative convergence analysis of the PSD by Neymeyr under much weaker assumptions. We embed Neymeyr's approach into the analysis of the PSD-id using a restricted formulation of the PSD-id. More importantly, we extend the new convergence analysis of the PSD-id to a practically preferred block version of the PSD-id, or BPSD-id, and show the cluster robustness of the BPSD-id. Numerical examples are provided to validate the theoretical estimates.
△ Less
Submitted 7 September, 2022;
originally announced September 2022.
-
Convergence rates of individual Ritz values in block preconditioned gradient-type eigensolvers
Authors:
Ming Zhou,
Klaus Neymeyr
Abstract:
Many popular eigensolvers for large and sparse Hermitian matrices or matrix pairs can be interpreted as accelerated block preconditioned gradient (BPG) iterations in order to analyze their convergence behavior by composing known estimates. An important feature of BPG is the cluster robustness, i.e., reasonable performance for computing clustered eigenvalues is ensured by a sufficiently large block…
▽ More
Many popular eigensolvers for large and sparse Hermitian matrices or matrix pairs can be interpreted as accelerated block preconditioned gradient (BPG) iterations in order to analyze their convergence behavior by composing known estimates. An important feature of BPG is the cluster robustness, i.e., reasonable performance for computing clustered eigenvalues is ensured by a sufficiently large block size. This feature can easily be explained for exact-inverse (exact shift-inverse) preconditioning by adapting classical estimates on nonpreconditioned eigensolvers, whereas the existing results for more general preconditioning are still improvable. We expect to extend certain sharp estimates for the corresponding vector iterations to BPG where proper bounds of convergence rates of individual Ritz values are to be derived. Such an extension has been achieved for BPG with fixed step sizes in [Math. Comp. 88 (2019), 2737--2765]. The present paper deals with the more practical case that the step sizes are implicitly optimized by the Rayleigh-Ritz method. Our new estimates improve some previous ones in view of concise and more flexible bounds.
△ Less
Submitted 1 June, 2022;
originally announced June 2022.
-
Majorization-type cluster robust bounds for block filters and eigensolvers
Authors:
M. Zhou,
M. E. Argentati,
A. V. Knyazev,
K. Neymeyr
Abstract:
Convergence analysis of block iterative solvers for Hermitian eigenvalue problems and the closely related research on properties of matrix-based signal filters are challenging, and attract increasing attention due to their recent applications in spectral data clustering and graph-based signal processing. We combine majorization-based techniques pioneered for investigating the Rayleigh-Ritz method…
▽ More
Convergence analysis of block iterative solvers for Hermitian eigenvalue problems and the closely related research on properties of matrix-based signal filters are challenging, and attract increasing attention due to their recent applications in spectral data clustering and graph-based signal processing. We combine majorization-based techniques pioneered for investigating the Rayleigh-Ritz method in [SIAM J. Matrix Anal. Appl., 31 (2010), pp. 1521-1537] with tools of classical analysis of the block power method by Rutishauser [Numer. Math., 13 (1969), pp. 4-13] to derive convergence rate bounds of an abstract block iteration, wherein tuples of tangents of principal angles or relative errors of Ritz values are bounded using majorization in terms of arranged partial sums and tuples of convergence factors. Our novel bounds are robust in presence of clusters of eigenvalues, improve some previous results, and are applicable to most known block iterative solvers and matrix-based filters, e.g., to block power, Chebyshev, and Lanczos methods combined with shift-and-invert approaches and polynomial filtering.
△ Less
Submitted 12 January, 2022;
originally announced January 2022.
-
Convergence theory for preconditioned eigenvalue solvers in a nutshell
Authors:
Merico E. Argentati,
Andrew V. Knyazev,
Klaus Neymeyr,
Evgueni E. Ovtchinnikov,
Ming Zhou
Abstract:
Preconditioned iterative methods for numerical solution of large matrix eigenvalue problems are increasingly gaining importance in various application areas, ranging from material sciences to data mining. Some of them, e.g., those using multilevel preconditioning for elliptic differential operators or graph Laplacian eigenvalue problems, exhibit almost optimal complexity in practice, i.e., their c…
▽ More
Preconditioned iterative methods for numerical solution of large matrix eigenvalue problems are increasingly gaining importance in various application areas, ranging from material sciences to data mining. Some of them, e.g., those using multilevel preconditioning for elliptic differential operators or graph Laplacian eigenvalue problems, exhibit almost optimal complexity in practice, i.e., their computational costs to calculate a fixed number of eigenvalues and eigenvectors grow linearly with the matrix problem size. Theoretical justification of their optimality requires convergence rate bounds that do not deteriorate with the increase of the problem size. Such bounds were pioneered by E. D'yakonov over three decades ago, but to date only a handful have been derived, mostly for symmetric eigenvalue problems. Just a few of known bounds are sharp. One of them is proved in [doi:10.1016/S0024-3795(01)00461-X] for the simplest preconditioned eigensolver with a fixed step size. The original proof has been greatly simplified and shortened in [doi:10.1137/080727567] by using a gradient flow integration approach. In the present work, we give an even more succinct proof, using novel ideas based on Karush-Kuhn-Tucker theory and nonlinear programming.
△ Less
Submitted 6 November, 2015; v1 submitted 16 December, 2014;
originally announced December 2014.
-
A geometric convergence theory for the preconditioned steepest descent iteration
Authors:
Klaus Neymeyr
Abstract:
Preconditioned gradient iterations for very large eigenvalue problems are efficient solvers with growing popularity. However, only for the simplest preconditioned eigensolver, namely the preconditioned gradient iteration (or preconditioned inverse iteration) with fixed step size, sharp non-asymptotic convergence estimates are known and these estimates require an ideally scaled preconditioner. In t…
▽ More
Preconditioned gradient iterations for very large eigenvalue problems are efficient solvers with growing popularity. However, only for the simplest preconditioned eigensolver, namely the preconditioned gradient iteration (or preconditioned inverse iteration) with fixed step size, sharp non-asymptotic convergence estimates are known and these estimates require an ideally scaled preconditioner. In this paper a new sharp convergence estimate is derived for the preconditioned steepest descent iteration which combines the preconditioned gradient iteration with the Rayleigh-Ritz procedure for optimal line search convergence acceleration. The new estimate always improves that of the fixed step size iteration. The practical importance of this new estimate is that arbitrarily scaled preconditioners can be used. The Rayleigh-Ritz procedure implicitly computes the optimal scaling.
△ Less
Submitted 11 August, 2011;
originally announced August 2011.
-
Gradient flow approach to geometric convergence analysis of preconditioned eigensolvers
Authors:
Andrew V. Knyazev,
Klaus Neymeyr
Abstract:
Preconditioned eigenvalue solvers (eigensolvers) are gaining popularity, but their convergence theory remains sparse and complex. We consider the simplest preconditioned eigensolver--the gradient iterative method with a fixed step size--for symmetric generalized eigenvalue problems, where we use the gradient of the Rayleigh quotient as an optimization direction. A sharp convergence rate bound fo…
▽ More
Preconditioned eigenvalue solvers (eigensolvers) are gaining popularity, but their convergence theory remains sparse and complex. We consider the simplest preconditioned eigensolver--the gradient iterative method with a fixed step size--for symmetric generalized eigenvalue problems, where we use the gradient of the Rayleigh quotient as an optimization direction. A sharp convergence rate bound for this method has been obtained in 2001--2003. It still remains the only known such bound for any of the methods in this class. While the bound is short and simple, its proof is not. We extend the bound to Hermitian matrices in the complex space and present a new self-contained and significantly shorter proof using novel geometric ideas.
△ Less
Submitted 16 March, 2009; v1 submitted 20 January, 2008;
originally announced January 2008.
