Showing 1–26 of 26 results for author: Eftekhari, A

Building Interpretable Climate Emulators for Economics

Authors: Aryan Eftekhari, Doris Folini, Aleksandra Friedl, Felix Kübler, Simon Scheidegger, Olaf Schenk

Abstract: We introduce a framework for developing efficient and interpretable climate emulators (CEs) for economic models of climate change. The paper makes two main contributions. First, we propose a general framework for constructing carbon-cycle emulators (CCEs) for macroeconomic models. The framework is implemented as a generalized linear multi-reservoir (box) model that conserves key physical quantitie… ▽ More We introduce a framework for developing efficient and interpretable climate emulators (CEs) for economic models of climate change. The paper makes two main contributions. First, we propose a general framework for constructing carbon-cycle emulators (CCEs) for macroeconomic models. The framework is implemented as a generalized linear multi-reservoir (box) model that conserves key physical quantities and can be customized for specific applications. We consider three versions of the CCE, which we evaluate within a simple representative agent economic model: (i) a three-box setting comparable to DICE-2016, (ii) a four-box extension, and (iii) a four-box version that explicitly captures land-use change. While the three-box model reproduces benchmark results well and the fourth reservoir adds little, incorporating the impact of land-use change on the carbon storage capacity of the terrestrial biosphere substantially alters atmospheric carbon stocks, temperature trajectories, and the optimal mitigation path. Second, we investigate pattern-scaling techniques that transform global-mean temperature projections from CEs into spatially heterogeneous warming fields. We show how regional baseline climates, non-uniform warming, and the associated uncertainties propagate into economic damages. △ Less

Submitted 13 June, 2025; v1 submitted 16 November, 2024; originally announced November 2024.

High-Dimensional Dynamic Stochastic Model Representation

Authors: Aryan Eftekhari, Simon Scheidegger

Abstract: We propose a scalable method for computing global solutions of nonlinear, high-dimensional dynamic stochastic economic models. First, within a time iteration framework, we approximate economic policy functions using an adaptive, high-dimensional model representation scheme, combined with adaptive sparse grids to address the ubiquitous challenge of the curse of dimensionality. Moreover, the adaptiv… ▽ More We propose a scalable method for computing global solutions of nonlinear, high-dimensional dynamic stochastic economic models. First, within a time iteration framework, we approximate economic policy functions using an adaptive, high-dimensional model representation scheme, combined with adaptive sparse grids to address the ubiquitous challenge of the curse of dimensionality. Moreover, the adaptivity within the individual component functions increases sparsity since grid points are added only where they are most needed, that is, in regions with steep gradients or at nondifferentiabilities. Second, we introduce a performant vectorization scheme for the interpolation compute kernel. Third, the algorithm is hybrid parallelized, leveraging both distributed- and shared-memory architectures. We observe significant speedups over the state-of-the-art techniques, and almost ideal strong scaling up to at least $1,000$ compute nodes of a Cray XC$50$ system at the Swiss National Supercomputing Center. Finally, to demonstrate our method's broad applicability, we compute global solutions to two variates of a high-dimensional international real business cycle model up to $300$ continuous state variables. In addition, we highlight a complementary advantage of the framework, which allows for a priori analysis of the model complexity. △ Less

Submitted 14 February, 2022; originally announced February 2022.

Over-Parametrized Matrix Factorization in the Presence of Spurious Stationary Points

Authors: Armin Eftekhari

Abstract: Motivated by the emerging role of interpolating machines in signal processing and machine learning, this work considers the computational aspects of over-parametrized matrix factorization. In this context, the optimization landscape may contain spurious stationary points (SSPs), which are proved to be full-rank matrices. The presence of these SSPs means that it is impossible to hope for any global… ▽ More Motivated by the emerging role of interpolating machines in signal processing and machine learning, this work considers the computational aspects of over-parametrized matrix factorization. In this context, the optimization landscape may contain spurious stationary points (SSPs), which are proved to be full-rank matrices. The presence of these SSPs means that it is impossible to hope for any global guarantees in over-parametrized matrix factorization. For example, when initialized at an SSP, the gradient flow will be trapped there forever. Nevertheless, despite these SSPs, we establish in this work that the gradient flow of the corresponding merit function converges to a global minimizer, provided that its initialization is rank-deficient and sufficiently close to the feasible set of the optimization problem. We numerically observe that a heuristic discretization of the proposed gradient flow, inspired by primal-dual algorithms, is successful when initialized randomly. Our result is in sharp contrast with the local refinement methods which require an initialization close to the optimal set of the optimization problem. More specifically, we successfully avoid the traps set by the SSPs because the gradient flow remains rank-deficient at all times, and not because there are no SSPs nearby. The latter is the case for the local refinement methods. Moreover, the widely-used restricted isometry property plays no role in our main result. △ Less

Submitted 25 December, 2021; originally announced December 2021.

Subquadratic Overparameterization for Shallow Neural Networks

Authors: Chaehwan Song, Ali Ramezani-Kebrya, Thomas Pethick, Armin Eftekhari, Volkan Cevher

Abstract: Overparameterization refers to the important phenomenon where the width of a neural network is chosen such that learning algorithms can provably attain zero loss in nonconvex training. The existing theory establishes such global convergence using various initialization strategies, training modifications, and width scalings. In particular, the state-of-the-art results require the width to scale qua… ▽ More Overparameterization refers to the important phenomenon where the width of a neural network is chosen such that learning algorithms can provably attain zero loss in nonconvex training. The existing theory establishes such global convergence using various initialization strategies, training modifications, and width scalings. In particular, the state-of-the-art results require the width to scale quadratically with the number of training data under standard initialization strategies used in practice for best generalization performance. In contrast, the most recent results obtain linear scaling either with requiring initializations that lead to the "lazy-training", or training only a single layer. In this work, we provide an analytical framework that allows us to adopt standard initialization strategies, possibly avoid lazy training, and train all layers simultaneously in basic shallow neural networks while attaining a desirable subquadratic scaling on the network width. We achieve the desiderata via Polyak-Lojasiewicz condition, smoothness, and standard assumptions on data, and use tools from random matrix theory. △ Less

Submitted 2 November, 2021; originally announced November 2021.

Comments: To appear at the conference on Neural Information Processing Systems (NeurIPS 2021)

Nonlinear matrix recovery using optimization on the Grassmann manifold

Authors: Florentin Goyens, Coralia Cartis, Armin Eftekhari

Abstract: We investigate the problem of recovering a partially observed high-rank matrix whose columns obey a nonlinear structure such as a union of subspaces, an algebraic variety or grouped in clusters. The recovery problem is formulated as the rank minimization of a nonlinear feature map applied to the original matrix, which is then further approximated by a constrained non-convex optimization problem in… ▽ More We investigate the problem of recovering a partially observed high-rank matrix whose columns obey a nonlinear structure such as a union of subspaces, an algebraic variety or grouped in clusters. The recovery problem is formulated as the rank minimization of a nonlinear feature map applied to the original matrix, which is then further approximated by a constrained non-convex optimization problem involving the Grassmann manifold. We propose two sets of algorithms, one arising from Riemannian optimization and the other as an alternating minimization scheme, both of which include first- and second-order variants. Both sets of algorithms have theoretical guarantees. In particular, for the alternating minimization, we establish global convergence and worst-case complexity bounds. Additionally, using the Kurdyka-Lojasiewicz property, we show that the alternating minimization converges to a unique limit point. We provide extensive numerical results for the recovery of union of subspaces and clustering under entry sampling and dense Gaussian sampling. Our methods are competitive with existing approaches and, in particular, high accuracy is achieved in the recovery using Riemannian second-order methods. △ Less

Submitted 8 December, 2022; v1 submitted 13 September, 2021; originally announced September 2021.

Comments: Fixed some typos in version 2

Principal Component Hierarchy for Sparse Quadratic Programs

Authors: Robbie Vreugdenhil, Viet Anh Nguyen, Armin Eftekhari, Peyman Mohajerin Esfahani

Abstract: We propose a novel approximation hierarchy for cardinality-constrained, convex quadratic programs that exploits the rank-dominating eigenvectors of the quadratic matrix. Each level of approximation admits a min-max characterization whose objective function can be optimized over the binary variables analytically, while preserving convexity in the continuous variables. Exploiting this property, we p… ▽ More We propose a novel approximation hierarchy for cardinality-constrained, convex quadratic programs that exploits the rank-dominating eigenvectors of the quadratic matrix. Each level of approximation admits a min-max characterization whose objective function can be optimized over the binary variables analytically, while preserving convexity in the continuous variables. Exploiting this property, we propose two scalable optimization algorithms, coined as the "best response" and the "dual program", that can efficiently screen the potential indices of the nonzero elements of the original program. We show that the proposed methods are competitive with the existing screening methods in the current sparse regression literature, and it is particularly fast on instances with high number of measurements in experiments with both synthetic and real datasets. △ Less

Submitted 25 May, 2021; originally announced May 2021.

Journal ref: ICML 2021

The Nonconvex Geometry of Linear Inverse Problems

Authors: Armin Eftekhari, Peyman Mohajerin Esfahani

Abstract: The gauge function, closely related to the atomic norm, measures the complexity of a statistical model, and has found broad applications in machine learning and statistical signal processing. In a high-dimensional learning problem, the gauge function attempts to safeguard against overfitting by promoting a sparse (concise) representation within the learning alphabet. In this work, within the con… ▽ More The gauge function, closely related to the atomic norm, measures the complexity of a statistical model, and has found broad applications in machine learning and statistical signal processing. In a high-dimensional learning problem, the gauge function attempts to safeguard against overfitting by promoting a sparse (concise) representation within the learning alphabet. In this work, within the context of linear inverse problems, we pinpoint the source of its success, but also argue that the applicability of the gauge function is inherently limited by its convexity, and showcase several learning problems where the classical gauge function theory fails. We then introduce a new notion of statistical complexity, gauge$_p$ function, which overcomes the limitations of the gauge function. The gauge$_p$ function is a simple generalization of the gauge function that can tightly control the sparsity of a statistical model within the learning alphabet and, perhaps surprisingly, draws further inspiration from the Burer-Monteiro factorization in computational mathematics. We also propose a new learning machine, with the building block of gauge$_p$ function, and arm this machine with a number of statistical guarantees. The potential of the proposed gauge$_p$ function theory is then studied for two stylized applications. Finally, we discuss the computational aspects and, in particular, suggest a tractable numerical algorithm for implementing the new learning machine. △ Less

Submitted 9 March, 2022; v1 submitted 7 January, 2021; originally announced January 2021.

Limitations of Implicit Bias in Matrix Sensing: Initialization Rank Matters

Authors: Armin Eftekhari, Konstantinos Zygalakis

Abstract: In matrix sensing, we first numerically identify the sensitivity to the initialization rank as a new limitation of the implicit bias of gradient flow. We will partially quantify this phenomenon mathematically, where we establish that the gradient flow of the empirical risk is implicitly biased towards low-rank outcomes and successfully learns the planted low-rank matrix, provided that the initiali… ▽ More In matrix sensing, we first numerically identify the sensitivity to the initialization rank as a new limitation of the implicit bias of gradient flow. We will partially quantify this phenomenon mathematically, where we establish that the gradient flow of the empirical risk is implicitly biased towards low-rank outcomes and successfully learns the planted low-rank matrix, provided that the initialization is low-rank and within a specific "capture neighborhood". This capture neighborhood is far larger than the corresponding neighborhood in local refinement results; the former contains all models with zero training error whereas the latter is a small neighborhood of a model with zero test error. These new insights enable us to design an alternative algorithm for matrix sensing that complements the high-rank and near-zero initialization scheme which is predominant in the existing literature. △ Less

Submitted 6 June, 2021; v1 submitted 27 August, 2020; originally announced August 2020.

Double-Loop Unadjusted Langevin Algorithm

Authors: Paul Rolland, Armin Eftekhari, Ali Kavis, Volkan Cevher

Abstract: A well-known first-order method for sampling from log-concave probability distributions is the Unadjusted Langevin Algorithm (ULA). This work proposes a new annealing step-size schedule for ULA, which allows to prove new convergence guarantees for sampling from a smooth log-concave distribution, which are not covered by existing state-of-the-art convergence guarantees. To establish this result, we… ▽ More A well-known first-order method for sampling from log-concave probability distributions is the Unadjusted Langevin Algorithm (ULA). This work proposes a new annealing step-size schedule for ULA, which allows to prove new convergence guarantees for sampling from a smooth log-concave distribution, which are not covered by existing state-of-the-art convergence guarantees. To establish this result, we derive a new theoretical bound that relates the Wasserstein distance to total variation distance between any two log-concave distributions that complements the reach of Talagrand T2 inequality. Moreover, applying this new step size schedule to an existing constrained sampling algorithm, we show state-of-the-art convergence rates for sampling from a constrained log-concave distribution, as well as improved dimension dependence. △ Less

Submitted 2 July, 2020; originally announced July 2020.

Nearly Minimal Over-Parametrization of Shallow Neural Networks

Authors: Armin Eftekhari, ChaeHwan Song, Volkan Cevher

Abstract: A recent line of work has shown that an overparametrized neural network can perfectly fit the training data, an otherwise often intractable nonconvex optimization problem. For (fully-connected) shallow networks, in the best case scenario, the existing theory requires quadratic over-parametrization as a function of the number of training samples. This paper establishes that linear overparametrizati… ▽ More A recent line of work has shown that an overparametrized neural network can perfectly fit the training data, an otherwise often intractable nonconvex optimization problem. For (fully-connected) shallow networks, in the best case scenario, the existing theory requires quadratic over-parametrization as a function of the number of training samples. This paper establishes that linear overparametrization is sufficient to fit the training data, using a simple variant of the (stochastic) gradient descent. Crucially, unlike several related works, the training considered in this paper is not limited to the lazy regime in the sense cautioned against in [1, 2]. Beyond shallow networks, the framework developed in this work for over-parametrization is applicable to a variety of learning problems. △ Less

Submitted 29 October, 2019; v1 submitted 9 October, 2019; originally announced October 2019.

Comments: This paper is submitted without consent of the co-authors

Fast and Provable ADMM for Learning with Generative Priors

Authors: Fabian Latorre Gómez, Armin Eftekhari, Volkan Cevher

Abstract: In this work, we propose a (linearized) Alternating Direction Method-of-Multipliers (ADMM) algorithm for minimizing a convex function subject to a nonconvex constraint. We focus on the special case where such constraint arises from the specification that a variable should lie in the range of a neural network. This is motivated by recent successful applications of Generative Adversarial Networks (G… ▽ More In this work, we propose a (linearized) Alternating Direction Method-of-Multipliers (ADMM) algorithm for minimizing a convex function subject to a nonconvex constraint. We focus on the special case where such constraint arises from the specification that a variable should lie in the range of a neural network. This is motivated by recent successful applications of Generative Adversarial Networks (GANs) in tasks like compressive sensing, denoising and robustness against adversarial examples. The derived rates for our algorithm are characterized in terms of certain geometric properties of the generator network, which we show hold for feedforward architectures, under mild assumptions. Unlike gradient descent (GD), it can efficiently handle non-smooth objectives as well as exploit efficient partial minimization procedures, thus being faster in many practical scenarios. △ Less

Submitted 7 July, 2019; originally announced July 2019.

Scalable Learning-Based Sampling Optimization for Compressive Dynamic MRI

Authors: Thomas Sanchez, Baran Gözcü, Ruud B. van Heeswijk, Armin Eftekhari, Efe Ilıcak, Tolga Çukur, Volkan Cevher

Abstract: Compressed sensing applied to magnetic resonance imaging (MRI) allows to reduce the scanning time by enabling images to be reconstructed from highly undersampled data. In this paper, we tackle the problem of designing a sampling mask for an arbitrary reconstruction method and a limited acquisition budget. Namely, we look for an optimal probability distribution from which a mask with a fixed cardin… ▽ More Compressed sensing applied to magnetic resonance imaging (MRI) allows to reduce the scanning time by enabling images to be reconstructed from highly undersampled data. In this paper, we tackle the problem of designing a sampling mask for an arbitrary reconstruction method and a limited acquisition budget. Namely, we look for an optimal probability distribution from which a mask with a fixed cardinality is drawn. We demonstrate that this problem admits a compactly supported solution, which leads to a deterministic optimal sampling mask. We then propose a stochastic greedy algorithm that (i) provides an approximate solution to this problem, and (ii) resolves the scaling issues of [1,2]. We validate its performance on in vivo dynamic MRI with retrospective undersampling, showing that our method preserves the performance of [1,2] while reducing the computational burden by a factor close to 200. △ Less

Submitted 16 March, 2020; v1 submitted 1 February, 2019; originally announced February 2019.

Comments: 13 pages, 16 figures, ICASSP 2020 - Session on "Learning and Optimization in Non-Convex Environments". Code available at https://github.com/t-sanchez/stochasticGreedyMRI.git

MOSES: A Streaming Algorithm for Linear Dimensionality Reduction

Authors: Armin Eftekhari, Raphael A. Hauser, Andreas Grammenos

Abstract: This paper introduces Memory-limited Online Subspace Estimation Scheme (MOSES) for both estimating the principal components of streaming data and reducing its dimension. More specifically, in various applications such as sensor networks, the data vectors are presented sequentially to a user who has limited storage and processing time available. Applied to such problems, MOSES can provide a running… ▽ More This paper introduces Memory-limited Online Subspace Estimation Scheme (MOSES) for both estimating the principal components of streaming data and reducing its dimension. More specifically, in various applications such as sensor networks, the data vectors are presented sequentially to a user who has limited storage and processing time available. Applied to such problems, MOSES can provide a running estimate of leading principal components of the data that has arrived so far and also reduce its dimension. MOSES generalises the popular incremental Singular Vale Decomposition (iSVD) to handle thin blocks of data, rather than just vectors. This minor generalisation in part allows us to complement MOSES with a comprehensive statistical analysis, thus providing the first theoretically-sound variant of iSVD, which has been lacking despite the empirical success of this method. This generalisation also enables us to concretely interpret MOSES as an approximate solver for the underlying non-convex optimisation program. We find that MOSES consistently surpasses the state of the art in our numerical experiments with both synthetic and real-world datasets, while being computationally inexpensive. △ Less

Submitted 23 February, 2020; v1 submitted 4 June, 2018; originally announced June 2018.

Stable Super-Resolution of Images: A Theoretical Study

Authors: Armin Eftekhari, Tamir Bendory, Gongguo Tang

Abstract: We study the ubiquitous super-resolution problem, in which one aims at localizing positive point sources in an image, blurred by the point spread function of the imaging device. To recover the point sources, we propose to solve a convex feasibility program, which simply finds a nonnegative Borel measure that agrees with the observations collected by the imaging device. In the absence of imaging… ▽ More We study the ubiquitous super-resolution problem, in which one aims at localizing positive point sources in an image, blurred by the point spread function of the imaging device. To recover the point sources, we propose to solve a convex feasibility program, which simply finds a nonnegative Borel measure that agrees with the observations collected by the imaging device. In the absence of imaging noise, we show that solving this convex program uniquely retrieves the point sources, provided that the imaging device collects enough observations. This result holds true if the point spread function of the imaging device can be decomposed into horizontal and vertical components, and if the translations of these components form a Chebyshev system, i.e., a system of continuous functions that loosely behave like algebraic polynomials. Building upon recent results for one-dimensional signals [1], we prove that this super-resolution algorithm is stable, in the generalized Wasserstein metric, to model mismatch (i.e., when the image is not sparse) and to additive imaging noise. In particular, the recovery error depends on the noise level and how well the image can be approximated with well-separated point sources. As an example, we verify these claims for the important case of a Gaussian point spread function. The proofs rely on the construction of novel interpolating polynomials---which are the main technical contribution of this paper---and partially resolve the question raised in [2] about the extension of the standard machinery to higher dimensions. △ Less

Submitted 5 September, 2020; v1 submitted 24 May, 2018; originally announced May 2018.

Sparse non-negative super-resolution -- simplified and stabilised

Authors: Armin Eftekhari, Jared Tanner, Andrew Thompson, Bogdan Toader, Hemant Tyagi

Abstract: The convolution of a discrete measure, $x=\sum_{i=1}^ka_iδ_{t_i}$, with a local window function, $φ(s-t)$, is a common model for a measurement device whose resolution is substantially lower than that of the objects being observed. Super-resolution concerns localising the point sources $\{a_i,t_i\}_{i=1}^k$ with an accuracy beyond the essential support of $φ(s-t)$, typically from $m$ samples… ▽ More The convolution of a discrete measure, $x=\sum_{i=1}^ka_iδ_{t_i}$, with a local window function, $φ(s-t)$, is a common model for a measurement device whose resolution is substantially lower than that of the objects being observed. Super-resolution concerns localising the point sources $\{a_i,t_i\}_{i=1}^k$ with an accuracy beyond the essential support of $φ(s-t)$, typically from $m$ samples $y(s_j)=\sum_{i=1}^k a_iφ(s_j-t_i)+η_j$, where $η_j$ indicates an inexactness in the sample value. We consider the setting of $x$ being non-negative and seek to characterise all non-negative measures approximately consistent with the samples. We first show that $x$ is the unique non-negative measure consistent with the samples provided the samples are exact, i.e. $η_j=0$, $m\ge 2k+1$ samples are available, and $φ(s-t)$ generates a Chebyshev system. This is independent of how close the sample locations are and {\em does not rely on any regulariser beyond non-negativity}; as such, it extends and clarifies the work by Schiebinger et al. and De Castro et al., who achieve the same results but require a total variation regulariser, which we show is unnecessary. Moreover, we characterise non-negative solutions $\hat{x}$ consistent with the samples within the bound $\sum_{j=1}^mη_j^2\le δ^2$. Any such non-negative measure is within ${\mathcal O}(δ^{1/7})$ of the discrete measure $x$ generating the samples in the generalised Wasserstein distance, converging to one another as $δ$ approaches zero. We also show how to make these general results, for windows that form a Chebyshev system, precise for the case of $φ(s-t)$ being a Gaussian window. The main innovation of these results is that non-negativity alone is sufficient to localise point sources beyond the essential sensor resolution. △ Less

Submitted 26 November, 2019; v1 submitted 4 April, 2018; originally announced April 2018.

Comments: 59 pages, 7 figures

Randomized Learning of the Second-Moment Matrix of a Smooth Function

Authors: Armin Eftekhari, Michael B. Wakin, Ping Li, Paul G. Constantine

Abstract: Consider an open set $\mathbb{D}\subseteq\mathbb{R}^n$, equipped with a probability measure $μ$. An important characteristic of a smooth function $f:\mathbb{D}\rightarrow\mathbb{R}$ is its \emph{second-moment matrix} $Σ_μ:=\int \nabla f(x) \nabla f(x)^* μ(dx) \in\mathbb{R}^{n\times n}$, where $\nabla f(x)\in\mathbb{R}^n$ is the gradient of $f(\cdot)$ at $x\in\mathbb{D}$ and $*$ stands for transpos… ▽ More Consider an open set $\mathbb{D}\subseteq\mathbb{R}^n$, equipped with a probability measure $μ$. An important characteristic of a smooth function $f:\mathbb{D}\rightarrow\mathbb{R}$ is its \emph{second-moment matrix} $Σ_μ:=\int \nabla f(x) \nabla f(x)^* μ(dx) \in\mathbb{R}^{n\times n}$, where $\nabla f(x)\in\mathbb{R}^n$ is the gradient of $f(\cdot)$ at $x\in\mathbb{D}$ and $*$ stands for transpose. For instance, the span of the leading $r$ eigenvectors of $Σ_μ$ forms an \emph{active subspace} of $f(\cdot)$, which contains the directions along which $f(\cdot)$ changes the most and is of particular interest in \emph{ridge approximation}. In this work, we propose a simple algorithm for estimating $Σ_μ$ from random point evaluations of $f(\cdot)$ \emph{without} imposing any structural assumptions on $Σ_μ$. Theoretical guarantees for this algorithm are established with the aid of the same technical tools that have proved valuable in the context of covariance matrix estimation from partial measurements. △ Less

Submitted 8 September, 2019; v1 submitted 19 December, 2016; originally announced December 2016.

Weighted Matrix Completion and Recovery with Prior Subspace Information

Authors: Armin Eftekhari, Dehui Yang, Michael B. Wakin

Abstract: An incoherent low-rank matrix can be efficiently reconstructed after observing a few of its entries at random, and then solving a convex program that minimizes the nuclear norm. In many applications, in addition to these entries, potentially valuable prior knowledge about the column and row spaces of the matrix is also available to the practitioner. In this paper, we incorporate this prior knowled… ▽ More An incoherent low-rank matrix can be efficiently reconstructed after observing a few of its entries at random, and then solving a convex program that minimizes the nuclear norm. In many applications, in addition to these entries, potentially valuable prior knowledge about the column and row spaces of the matrix is also available to the practitioner. In this paper, we incorporate this prior knowledge in matrix completion---by minimizing a weighted nuclear norm---and precisely quantify any improvements. In particular, we find in theory that reliable prior knowledge reduces the sample complexity of matrix completion by a logarithmic factor, and the observed improvement in numerical simulations is considerably more magnified. We also present similar results for the closely related problem of matrix recovery from generic linear measurements. △ Less

Submitted 12 March, 2018; v1 submitted 6 December, 2016; originally announced December 2016.

Streaming Principal Component Analysis From Incomplete Data

Authors: Armin Eftekhari, Gregory Ongie, Laura Balzano, Michael B. Wakin

Abstract: Linear subspace models are pervasive in computational sciences and particularly used for large datasets which are often incomplete due to privacy issues or sampling constraints. Therefore, a critical problem is developing an efficient algorithm for detecting low-dimensional linear structure from incomplete data efficiently, in terms of both computational complexity and storage. In this paper we… ▽ More Linear subspace models are pervasive in computational sciences and particularly used for large datasets which are often incomplete due to privacy issues or sampling constraints. Therefore, a critical problem is developing an efficient algorithm for detecting low-dimensional linear structure from incomplete data efficiently, in terms of both computational complexity and storage. In this paper we propose a streaming subspace estimation algorithm called Subspace Navigation via Interpolation from Partial Entries (SNIPE) that efficiently processes blocks of incomplete data to estimate the underlying subspace model. In every iteration, SNIPE finds the subspace that best fits the new data block but remains close to the previous estimate. We show that SNIPE is a streaming solver for the underlying nonconvex matrix completion problem, that it converges globally to a stationary point of this program regardless of initialization, and that the convergence is locally linear with high probability. We also find that SNIPE shows state-of-the-art performance in our numerical simulations. △ Less

Submitted 21 May, 2018; v1 submitted 2 December, 2016; originally announced December 2016.

What to Expect When You Are Expecting on the Grassmannian

Authors: Armin Eftekhari, Laura Balzano, Michael B. Wakin

Abstract: Consider an incoming sequence of vectors, all belonging to an unknown subspace $\operatorname{S}$, and each with many missing entries. In order to estimate $\operatorname{S}$, it is common to partition the data into blocks and iteratively update the estimate of $\operatorname{S}$ with each new incoming measurement block. In this paper, we investigate a rather basic question: Is it possible to id… ▽ More Consider an incoming sequence of vectors, all belonging to an unknown subspace $\operatorname{S}$, and each with many missing entries. In order to estimate $\operatorname{S}$, it is common to partition the data into blocks and iteratively update the estimate of $\operatorname{S}$ with each new incoming measurement block. In this paper, we investigate a rather basic question: Is it possible to identify $\operatorname{S}$ by averaging the column span of the partially observed incoming measurement blocks on the Grassmannian? We show that in general the span of the incoming blocks is in fact a biased estimator of $\operatorname{S}$ when data suffers from erasures, and we find an upper bound for this bias. We reach this conclusion by examining the defining optimization program for the Fréchet expectation on the Grassmannian, and with the aid of a sharp perturbation bound and standard large deviation results. △ Less

Submitted 5 March, 2017; v1 submitted 22 November, 2016; originally announced November 2016.

MC^2: A Two-Phase Algorithm for Leveraged Matrix Completion

Authors: Armin Eftekhari, Michael B. Wakin, Rachel A. Ward

Abstract: Leverage scores, loosely speaking, reflect the importance of the rows and columns of a matrix. Ideally, given the leverage scores of a rank-$r$ matrix $M\in\mathbb{R}^{n\times n}$, that matrix can be reliably completed from just $O(rn\log^{2}n)$ samples if the samples are chosen randomly from a nonuniform distribution induced by the leverage scores. In practice, however, the leverage scores are of… ▽ More Leverage scores, loosely speaking, reflect the importance of the rows and columns of a matrix. Ideally, given the leverage scores of a rank-$r$ matrix $M\in\mathbb{R}^{n\times n}$, that matrix can be reliably completed from just $O(rn\log^{2}n)$ samples if the samples are chosen randomly from a nonuniform distribution induced by the leverage scores. In practice, however, the leverage scores are often unknown a priori. As such, the sample complexity in uniform matrix completion---using uniform random sampling---increases to $O(η(M)\cdot rn\log^{2}n)$, where $η(M)$ is the largest leverage score of $M$. In this paper, we propose a two-phase algorithm called MC$^2$ for matrix completion: in the first phase, the leverage scores are estimated based on uniform random samples, and then in the second phase the matrix is resampled nonuniformly based on the estimated leverage scores and then completed. For well-conditioned matrices, the total sample complexity of MC$^2$ is no worse than uniform matrix completion, and for certain classes of well-conditioned matrices---namely, reasonably coherent matrices whose leverage scores exhibit mild decay---MC$^2$ requires substantially fewer samples. Numerical simulations suggest that the algorithm outperforms uniform matrix completion in a broad class of matrices, and in particular, is much less sensitive to the condition number than our theory currently requires. △ Less

Submitted 17 November, 2017; v1 submitted 6 September, 2016; originally announced September 2016.

What Happens to a Manifold Under a Bi-Lipschitz Map?

Authors: Armin Eftekhari, Michael B. Wakin

Abstract: We study geometric and topological properties of the image of a smooth submanifold of $\mathbb{R}^{n}$ under a bi-Lipschitz map to $\mathbb{R}^{m}$. In particular, we characterize how the dimension, diameter, volume, and reach of the embedded manifold relate to the original. Our main result establishes a lower bound on the reach of the embedded manifold in the case where $m \le n$ and the bi-Lipsc… ▽ More We study geometric and topological properties of the image of a smooth submanifold of $\mathbb{R}^{n}$ under a bi-Lipschitz map to $\mathbb{R}^{m}$. In particular, we characterize how the dimension, diameter, volume, and reach of the embedded manifold relate to the original. Our main result establishes a lower bound on the reach of the embedded manifold in the case where $m \le n$ and the bi-Lipschitz map is linear. We discuss implications of this work in signal processing and machine learning, where bi-Lipschitz maps on low-dimensional manifolds have been constructed using randomized linear operators. △ Less

Submitted 21 November, 2016; v1 submitted 21 December, 2015; originally announced December 2015.

Greed is Super: A Fast Algorithm for Super-Resolution

Authors: Armin Eftekhari, Michael B. Wakin

Abstract: We present a fast two-phase algorithm for super-resolution with strong theoretical guarantees. Given the low-frequency part of the spectrum of a sequence of impulses, Phase I consists of a greedy algorithm that roughly estimates the impulse positions. These estimates are then refined by local optimization in Phase II. In contrast to the convex relaxation proposed by Candès et al., our approach h… ▽ More We present a fast two-phase algorithm for super-resolution with strong theoretical guarantees. Given the low-frequency part of the spectrum of a sequence of impulses, Phase I consists of a greedy algorithm that roughly estimates the impulse positions. These estimates are then refined by local optimization in Phase II. In contrast to the convex relaxation proposed by Candès et al., our approach has a low computational complexity but requires the impulses to be separated by an additional logarithmic factor to succeed. The backbone of our work is the fundamental work of Slepian et al. involving discrete prolate spheroidal wave functions and their unique properties. △ Less

Submitted 10 November, 2015; originally announced November 2015.

New Analysis of Manifold Embeddings and Signal Recovery from Compressive Measurements

Authors: Armin Eftekhari, Michael B. Wakin

Abstract: Compressive Sensing (CS) exploits the surprising fact that the information contained in a sparse signal can be preserved in a small number of compressive, often random linear measurements of that signal. Strong theoretical guarantees have been established concerning the embedding of a sparse signal family under a random measurement operator and on the accuracy to which sparse signals can be recove… ▽ More Compressive Sensing (CS) exploits the surprising fact that the information contained in a sparse signal can be preserved in a small number of compressive, often random linear measurements of that signal. Strong theoretical guarantees have been established concerning the embedding of a sparse signal family under a random measurement operator and on the accuracy to which sparse signals can be recovered from noisy compressive measurements. In this paper, we address similar questions in the context of a different modeling framework. Instead of sparse models, we focus on the broad class of manifold models, which can arise in both parametric and non-parametric signal families. Using tools from the theory of empirical processes, we improve upon previous results concerning the embedding of low-dimensional manifolds under random measurement operators. We also establish both deterministic and probabilistic instance-optimal bounds in $\ell_2$ for manifold-based signal recovery and parameter estimation from noisy compressive measurements. In line with analogous results for sparsity-based CS, we conclude that much stronger bounds are possible in the probabilistic setting. Our work supports the growing evidence that manifold-based models can be used with high accuracy in compressive signal processing. △ Less

Submitted 1 May, 2014; v1 submitted 19 June, 2013; originally announced June 2013.

Comments: arXiv admin note: substantial text overlap with arXiv:1002.1247

The Restricted Isometry Property for Random Block Diagonal Matrices

Authors: Armin Eftekhari, Han Lun Yap, Christopher J. Rozell, Michael B. Wakin

Abstract: In Compressive Sensing, the Restricted Isometry Property (RIP) ensures that robust recovery of sparse vectors is possible from noisy, undersampled measurements via computationally tractable algorithms. It is by now well-known that Gaussian (or, more generally, sub-Gaussian) random matrices satisfy the RIP under certain conditions on the number of measurements. Their use can be limited in practice,… ▽ More In Compressive Sensing, the Restricted Isometry Property (RIP) ensures that robust recovery of sparse vectors is possible from noisy, undersampled measurements via computationally tractable algorithms. It is by now well-known that Gaussian (or, more generally, sub-Gaussian) random matrices satisfy the RIP under certain conditions on the number of measurements. Their use can be limited in practice, however, due to storage limitations, computational considerations, or the mismatch of such matrices with certain measurement architectures. These issues have recently motivated considerable effort towards studying the RIP for structured random matrices. In this paper, we study the RIP for block diagonal measurement matrices where each block on the main diagonal is itself a sub-Gaussian random matrix. Our main result states that such matrices can indeed satisfy the RIP but that the requisite number of measurements depends on certain properties of the basis in which the signals are sparse. In the best case, these matrices perform nearly as well as dense Gaussian random matrices, despite having many fewer nonzero entries. △ Less

Submitted 13 February, 2014; v1 submitted 11 October, 2012; originally announced October 2012.

MSC Class: 94A20; 60B20; 46B09;

Matched Filtering from Limited Frequency Samples

Authors: Armin Eftekhari, Justin Romberg, Michael B. Wakin

Abstract: In this paper, we study a simple correlation-based strategy for estimating the unknown delay and amplitude of a signal based on a small number of noisy, randomly chosen frequency-domain samples. We model the output of this "compressive matched filter" as a random process whose mean equals the scaled, shifted autocorrelation function of the template signal. Using tools from the theory of empirical… ▽ More In this paper, we study a simple correlation-based strategy for estimating the unknown delay and amplitude of a signal based on a small number of noisy, randomly chosen frequency-domain samples. We model the output of this "compressive matched filter" as a random process whose mean equals the scaled, shifted autocorrelation function of the template signal. Using tools from the theory of empirical processes, we prove that the expected maximum deviation of this process from its mean decreases sharply as the number of measurements increases, and we also derive a probabilistic tail bound on the maximum deviation. Putting all of this together, we bound the minimum number of measurements required to guarantee that the empirical maximum of this random process occurs sufficiently close to the true peak of its mean function. We conclude that for broad classes of signals, this compressive matched filter will successfully estimate the unknown delay (with high probability, and within a prescribed tolerance) using a number of random frequency-domain samples that scales inversely with the signal-to-noise ratio and only logarithmically in the in the observation bandwidth and the possible range of delays. △ Less

Submitted 12 July, 2012; v1 submitted 13 January, 2011; originally announced January 2011.

Comments: Submitted to the IEEE Transactions on Information Theory on January 13, 2011

Fractal geometry of literature: first attempt to Shakespeare's works

Authors: Ali Eftekhari

Abstract: It was demonstrated that there is a geometrical order in the structure of literature. Fractal geometry as a modern mathematical approach and a new geometrical viewpoint on natural objects including both processes and structures was employed for analysis of literature. As the first study, the works of William Shakespeare were chosen as the most important items in western literature. By counting t… ▽ More It was demonstrated that there is a geometrical order in the structure of literature. Fractal geometry as a modern mathematical approach and a new geometrical viewpoint on natural objects including both processes and structures was employed for analysis of literature. As the first study, the works of William Shakespeare were chosen as the most important items in western literature. By counting the number of letters applied in a manuscript, it is possible to study the whole manuscript statistically. A novel method based on basic assumption of fractal geometry was proposed for the calculation of fractal dimensions of the literature. The results were compared with Zipf's law. Zipf's law was successfully used for letters instead of words. Two new concepts namely Zipf's dimension and Zipf's order were also introduced. It was found that changes of both fractal dimension and Zipf's dimension are similar and dependent on the manuscript length. Interestingly, direct plotting the data obtained in semi-logarithmic and logarithmic forms also led to a power-law. △ Less

Submitted 17 August, 2004; originally announced August 2004.

Comments: 26 pages, 7 figures, 3 tables