-
A New Complexity Result for Strongly Convex Optimization with Locally $α$-H{ö}lder Continuous Gradients
Authors:
Xiaojun Chen,
C. T. Kelley,
Lei Wang
Abstract:
In this paper, we present a new complexity result for the gradient descent method with an appropriately fixed stepsize for minimizing a strongly convex function with locally $α$-H{ö}lder continuous gradients ($0 < α\leq 1$). The complexity bound for finding an approximate minimizer with a distance to the true minimizer less than $\varepsilon$ is $O(\log (\varepsilon^{-1}) \varepsilon^{2 α- 2})$, w…
▽ More
In this paper, we present a new complexity result for the gradient descent method with an appropriately fixed stepsize for minimizing a strongly convex function with locally $α$-H{ö}lder continuous gradients ($0 < α\leq 1$). The complexity bound for finding an approximate minimizer with a distance to the true minimizer less than $\varepsilon$ is $O(\log (\varepsilon^{-1}) \varepsilon^{2 α- 2})$, which extends the well-known complexity result for $α= 1$.
△ Less
Submitted 6 May, 2025;
originally announced May 2025.
-
Interprecision transfers in iterative refinement
Authors:
C. T. Kelley
Abstract:
We make the interprecision transfers explicit in an algorithmic description of iterative refinement and obtain new insights into the algorithm. One example is the classic variant of iterative refinement where the matrix and the factorization are stored in a working precision and the residual is evaluated in a higher precision. In that case we make the observation that this algorithm will solve a p…
▽ More
We make the interprecision transfers explicit in an algorithmic description of iterative refinement and obtain new insights into the algorithm. One example is the classic variant of iterative refinement where the matrix and the factorization are stored in a working precision and the residual is evaluated in a higher precision. In that case we make the observation that this algorithm will solve a promoted form of the original problem and thereby characterize the limiting behavior in a novel way and obtain a different version of the classic convergence analysis. We also discuss two approaches for interprecision transfer in the triangular solves.
△ Less
Submitted 30 June, 2024;
originally announced July 2024.
-
Using MultiPrecisonArrays.jl: Iterative Refinement in Julia
Authors:
C. T. Kelley
Abstract:
MultiPrecisionArrays.jl is a Julia package. This package provides data structures and solvers for several variants of iterative refinement. It will become much more useful when half precision (aka Float16) is fully supported in LAPACK/BLAS. For now, its only general-purpose application is classical iterative refinement with double precision equations and single precision factorizations.
MultiPrecisionArrays.jl is a Julia package. This package provides data structures and solvers for several variants of iterative refinement. It will become much more useful when half precision (aka Float16) is fully supported in LAPACK/BLAS. For now, its only general-purpose application is classical iterative refinement with double precision equations and single precision factorizations.
△ Less
Submitted 30 September, 2024; v1 submitted 24 November, 2023;
originally announced November 2023.
-
Newton's Method in Three Precisions
Authors:
C. T. Kelley
Abstract:
We describe a three precision variant of Newton's method for nonlinear equations. We evaluate the nonlinear residual in double precision, store the Jacobian matrix in single precision, and solve the equation for the Newton step with iterative refinement with a factorization in half precision. We analyze the method as an inexact Newton method. This analysis shows that, except for very poorly condit…
▽ More
We describe a three precision variant of Newton's method for nonlinear equations. We evaluate the nonlinear residual in double precision, store the Jacobian matrix in single precision, and solve the equation for the Newton step with iterative refinement with a factorization in half precision. We analyze the method as an inexact Newton method. This analysis shows that, except for very poorly conditioned Jacobians, the number of nonlinear iterations needed is the same that one would get if one stored and factored the Jacobian in double precision. In many ill-conditioned cases one can use the low precision factorization as a preconditioner for a GMRES iteration. That approach can recover fast convergence of the nonlinear iteration. We present an example to illustrate the results.
△ Less
Submitted 9 October, 2023; v1 submitted 29 July, 2023;
originally announced July 2023.
-
Convolutional Autoencoders for Reduced-Order Modeling
Authors:
Sreeram Venkat,
Ralph C. Smith,
Carl T. Kelley
Abstract:
In the construction of reduced-order models for dynamical systems, linear projection methods, such as proper orthogonal decompositions, are commonly employed. However, for many dynamical systems, the lower dimensional representation of the state space can most accurately be described by a \textit{nonlinear} manifold. Previous research has shown that deep learning can provide an efficient method fo…
▽ More
In the construction of reduced-order models for dynamical systems, linear projection methods, such as proper orthogonal decompositions, are commonly employed. However, for many dynamical systems, the lower dimensional representation of the state space can most accurately be described by a \textit{nonlinear} manifold. Previous research has shown that deep learning can provide an efficient method for performing nonlinear dimension reduction, though they are dependent on the availability of training data and are often problem-specific \citep[see][]{carlberg_ca}. Here, we utilize randomized training data to create and train convolutional autoencoders that perform nonlinear dimension reduction for the wave and Kuramoto-Shivasinsky equations. Moreover, we present training methods that are independent of full-order model samples and use the manifold least-squares Petrov-Galerkin projection method to define a reduced-order model for the heat, wave, and Kuramoto-Shivasinsky equations using the same autoencoder.
△ Less
Submitted 27 August, 2021;
originally announced August 2021.
-
Condition Estimates for Pseudo-Arclength Continuation
Authors:
K. I. Dickson,
C. T. Kelley,
I. C. F. Ipsen,
I. G. Kevrekidis
Abstract:
We bound the condition number of the Jacobian in pseudo arclength continuation problems, and we quantify the effect of this condition number on the linear system solution in a Newton GMRES solve.
In pseudo arclength continuation one repeatedly solves systems of nonlinear equations $F(u(s),λ(s))=0$ for a real-valued function $u$ and a real parameter $λ$, given different values of the arclength…
▽ More
We bound the condition number of the Jacobian in pseudo arclength continuation problems, and we quantify the effect of this condition number on the linear system solution in a Newton GMRES solve.
In pseudo arclength continuation one repeatedly solves systems of nonlinear equations $F(u(s),λ(s))=0$ for a real-valued function $u$ and a real parameter $λ$, given different values of the arclength $s$. It is known that the Jacobian $F_x$ of $F$ with respect to $x=(u,λ)$ is nonsingular, if the path contains only regular points and simple fold singularities. We introduce a new characterization of simple folds in terms of the singular value decomposition, and we use it to derive a new bound for the norm of $F_x^{-1}$. We also show that the convergence rate of GMRES in a Newton step for $F(u(s),λ(s))=0$ is essentially the same as that of the original problem $G(u,λ)=0$. In particular we prove that the bounds on the degrees of the minimal polynomials of the Jacobians $F_x$ and $G_u$ differ by at most 2. We illustrate the effectiveness of our bounds with an example from radiative transfer theory.
△ Less
Submitted 30 March, 2006;
originally announced March 2006.
-
Newton-Krylov solvers for time-steppers
Authors:
C. T. Kelley,
I. G. Kevrekidis,
L. Qiao
Abstract:
We study how the Newton-GMRES iteration can enable dynamic simulators (time-steppers) to perform fixed-point and path-following computations.For a class of dissipative problems, whose dynamics are characterized by a slow manifold, the Jacobian matrices in such computations are compact perturbations of the identity. We examine the number of GMRES iterations required for each nonlinear iteration a…
▽ More
We study how the Newton-GMRES iteration can enable dynamic simulators (time-steppers) to perform fixed-point and path-following computations.For a class of dissipative problems, whose dynamics are characterized by a slow manifold, the Jacobian matrices in such computations are compact perturbations of the identity. We examine the number of GMRES iterations required for each nonlinear iteration as a function of the dimension of the slow subspace and the time-stepper reporting horizon. In a path-following computation, only a small number (one or two) of additional GMRES iterations is required.
△ Less
Submitted 20 April, 2004;
originally announced April 2004.
