-
The Trimmed Lasso: Sparsity and Robustness
Authors:
Dimitris Bertsimas,
Martin S. Copenhaver,
Rahul Mazumder
Abstract:
Nonconvex penalty methods for sparse modeling in linear regression have been a topic of fervent interest in recent years. Herein, we study a family of nonconvex penalty functions that we call the trimmed Lasso and that offers exact control over the desired level of sparsity of estimators. We analyze its structural properties and in doing so show the following:
1) Drawing parallels between robust…
▽ More
Nonconvex penalty methods for sparse modeling in linear regression have been a topic of fervent interest in recent years. Herein, we study a family of nonconvex penalty functions that we call the trimmed Lasso and that offers exact control over the desired level of sparsity of estimators. We analyze its structural properties and in doing so show the following:
1) Drawing parallels between robust statistics and robust optimization, we show that the trimmed-Lasso-regularized least squares problem can be viewed as a generalized form of total least squares under a specific model of uncertainty. In contrast, this same model of uncertainty, viewed instead through a robust optimization lens, leads to the convex SLOPE (or OWL) penalty.
2) Further, in relating the trimmed Lasso to commonly used sparsity-inducing penalty functions, we provide a succinct characterization of the connection between trimmed-Lasso- like approaches and penalty functions that are coordinate-wise separable, showing that the trimmed penalties subsume existing coordinate-wise separable penalties, with strict containment in general.
3) Finally, we describe a variety of exact and heuristic algorithms, both existing and new, for trimmed Lasso regularized estimation problems. We include a comparison between the different approaches and an accompanying implementation of the algorithms.
△ Less
Submitted 15 August, 2017;
originally announced August 2017.
-
Certifiably Optimal Low Rank Factor Analysis
Authors:
Dimitris Bertsimas,
Martin S. Copenhaver,
Rahul Mazumder
Abstract:
Factor Analysis (FA) is a technique of fundamental importance that is widely used in classical and modern multivariate statistics, psychometrics and econometrics. In this paper, we revisit the classical rank-constrained FA problem, which seeks to approximate an observed covariance matrix ($\boldsymbolΣ$), by the sum of a Positive Semidefinite (PSD) low-rank component ($\boldsymbolΘ$) and a diagona…
▽ More
Factor Analysis (FA) is a technique of fundamental importance that is widely used in classical and modern multivariate statistics, psychometrics and econometrics. In this paper, we revisit the classical rank-constrained FA problem, which seeks to approximate an observed covariance matrix ($\boldsymbolΣ$), by the sum of a Positive Semidefinite (PSD) low-rank component ($\boldsymbolΘ$) and a diagonal matrix ($\boldsymbolΦ$) (with nonnegative entries) subject to $\boldsymbolΣ- \boldsymbolΦ$ being PSD. We propose a flexible family of rank-constrained, nonlinear Semidefinite Optimization based formulations for this task. We introduce a reformulation of the problem as a smooth optimization problem with convex compact constraints and propose a unified algorithmic framework, utilizing state of the art techniques in nonlinear optimization to obtain high-quality feasible solutions for our proposed formulation. At the same time, by using a variety of techniques from discrete and global optimization, we show that these solutions are certifiably optimal in many cases, even for problems with thousands of variables. Our techniques are general and make no assumption on the underlying problem data. The estimator proposed herein, aids statistical interpretability, provides computational scalability and significantly improved accuracy when compared to current, publicly available popular methods for rank-constrained FA. We demonstrate the effectiveness of our proposal on an array of synthetic and real-life datasets. To our knowledge, this is the first paper that demonstrates how a previously intractable rank-constrained optimization problem can be solved to provable optimality by coupling developments in convex analysis and in discrete optimization.
△ Less
Submitted 22 April, 2016;
originally announced April 2016.
-
Characterization of the equivalence of robustification and regularization in linear and matrix regression
Authors:
Dimitris Bertsimas,
Martin S. Copenhaver
Abstract:
The notion of developing statistical methods in machine learning which are robust to adversarial perturbations in the underlying data has been the subject of increasing interest in recent years. A common feature of this work is that the adversarial robustification often corresponds exactly to regularization methods which appear as a loss function plus a penalty. In this paper we deepen and extend…
▽ More
The notion of developing statistical methods in machine learning which are robust to adversarial perturbations in the underlying data has been the subject of increasing interest in recent years. A common feature of this work is that the adversarial robustification often corresponds exactly to regularization methods which appear as a loss function plus a penalty. In this paper we deepen and extend the understanding of the connection between robustification and regularization (as achieved by penalization) in regression problems. Specifically, (a) in the context of linear regression, we characterize precisely under which conditions on the model of uncertainty used and on the loss function penalties robustification and regularization are equivalent, and (b) we extend the characterization of robustification and regularization to matrix regression problems (matrix completion and Principal Component Analysis).
△ Less
Submitted 25 February, 2017; v1 submitted 22 November, 2014;
originally announced November 2014.
-
On Structural Decompositions of Finite Frames
Authors:
Alice Z. -Y. Chan,
Martin S. Copenhaver,
Sivaram K. Narayan,
Logan Stokols,
Allison Theobold
Abstract:
A frame in an $n$-dimensional Hilbert space $H_n$ is a possibly redundant collection of vectors $\{f_i\}_{i\in I}$ that span the space. A tight frame is a generalization of an orthonormal basis. A frame $\{f_i\}_{i\in I}$ is said to be scalable if there exist nonnegative scalars $\{c_i\}_{i\in I}$ such that $\{c_if_i\}_{i\in I}$ is a tight frame. In this paper we study the combinatorial structure…
▽ More
A frame in an $n$-dimensional Hilbert space $H_n$ is a possibly redundant collection of vectors $\{f_i\}_{i\in I}$ that span the space. A tight frame is a generalization of an orthonormal basis. A frame $\{f_i\}_{i\in I}$ is said to be scalable if there exist nonnegative scalars $\{c_i\}_{i\in I}$ such that $\{c_if_i\}_{i\in I}$ is a tight frame. In this paper we study the combinatorial structure of frames and their decomposition into tight or scalable subsets by using partially-ordered sets (posets). We define the factor poset of a frame $\{f_i\}_{i\in I}$ to be a collection of subsets of $I$ ordered by inclusion so that nonempty $J\subseteq I$ is in the factor poset if and only if $\{f_j\}_{j\in J}$ is a tight frame for $H_n$. A similar definition is given for the scalability poset of a frame. We prove conditions which factor posets satisfy and use these to study the inverse factor poset problem, which inquires when there exists a frame whose factor poset is some given poset $P$. We determine a necessary condition for solving the inverse factor poset problem in $H_n$ which is also sufficient for $H_2$. We describe how factor poset structure of frames is preserved under orthogonal projections. We also consider the enumeration of the number of possible factor posets and bounds on the size of factors posets. We then turn our attention to scalable frames and present partial results regarding when a frame can be scaled to have a given factor poset.
△ Less
Submitted 22 November, 2014;
originally announced November 2014.
-
Factor posets of frames and dual frames in finite dimensions
Authors:
Kileen Berry,
Martin S. Copenhaver,
Eric Evert,
Yeon Hyang Kim,
Troy Klingler,
Sivaram K. Narayan,
Son T. Nghiem
Abstract:
We consider frames in a finite-dimensional Hilbert space where frames are exactly the spanning sets of the vector space. A factor poset of a frame is defined to be a collection of subsets of $I$, the index set of our vectors, ordered by inclusion so that nonempty $J \subseteq I$ is in the factor poset if and only if $\{f_i\}_{i \in J}$ is a tight frame. We first study when a poset…
▽ More
We consider frames in a finite-dimensional Hilbert space where frames are exactly the spanning sets of the vector space. A factor poset of a frame is defined to be a collection of subsets of $I$, the index set of our vectors, ordered by inclusion so that nonempty $J \subseteq I$ is in the factor poset if and only if $\{f_i\}_{i \in J}$ is a tight frame. We first study when a poset $P\subseteq 2^I$ is a factor poset of a frame and then relate the two topics by discussing the connections between the factor posets of frames and their duals. Additionally we discuss duals with regard to $\ell^p$ minimization.
△ Less
Submitted 15 November, 2014;
originally announced November 2014.
-
Maximum Robustness and Surgery of Frames in finite dimensions
Authors:
Martin S. Copenhaver,
Yeon Hyang Kim,
Cortney Logan,
Kyanne Mayfield,
Sivaram K. Narayan,
Jonathan Sheperd
Abstract:
We consider frames in a finite-dimensional Hilbert space Hn where frames are exactly the spanning sets of the vector space. We present a method to determine the maximum robustness of a frame. We present results on tight subframes and surgery of frames. We also answer the question of when length surgery resulting in a tight frame set for Hn is possible.
We consider frames in a finite-dimensional Hilbert space Hn where frames are exactly the spanning sets of the vector space. We present a method to determine the maximum robustness of a frame. We present results on tight subframes and surgery of frames. We also answer the question of when length surgery resulting in a tight frame set for Hn is possible.
△ Less
Submitted 5 March, 2013;
originally announced March 2013.
-
Diagram vectors and Tight Frame Scaling in Finite Dimensions
Authors:
Martin S. Copenhaver,
Yeon Hyang Kim,
Cortney Logan,
Kyanne Mayfield,
Sivaram K. Narayan,
Matthew J. Petro,
Jonathan Sheperd
Abstract:
We consider frames in a finite-dimensional Hilbert space Hn where frames are exactly the spanning sets of the vector space. The diagram vector of a vector in R2 was previously defined using polar coordinates and was used to characterize tight frames in R2 in a geometric fashion. Reformulating the definition of a diagram vector in R2 we provide a natural extension of this notion to Rn and Cn. Using…
▽ More
We consider frames in a finite-dimensional Hilbert space Hn where frames are exactly the spanning sets of the vector space. The diagram vector of a vector in R2 was previously defined using polar coordinates and was used to characterize tight frames in R2 in a geometric fashion. Reformulating the definition of a diagram vector in R2 we provide a natural extension of this notion to Rn and Cn. Using the diagram vectors we give a characterization of tight frames in Rn or Cn. Further we provide a characterization of when a unit-norm frame in Rn or Cn can be scaled to a tight frame. This classification allows us to determine all scaling coefficients that make a unit-norm frame into a tight frame.
△ Less
Submitted 5 March, 2013;
originally announced March 2013.
