Showing 1–5 of 5 results for author: Van De Geer, S A

The Smooth-Lasso and other $\ell_1+\ell_2$-penalized methods

Authors: Mohamed Hebiri, Sara A. Van De Geer

Abstract: We consider a linear regression problem in a high dimensional setting where the number of covariates $p$ can be much larger than the sample size $n$. In such a situation, one often assumes sparsity of the regression vector, \textit i.e., the regression vector contains many zero components. We propose a Lasso-type estimator $\hatβ^{Quad}$ (where '$Quad$' stands for quadratic) which is based on two… ▽ More We consider a linear regression problem in a high dimensional setting where the number of covariates $p$ can be much larger than the sample size $n$. In such a situation, one often assumes sparsity of the regression vector, \textit i.e., the regression vector contains many zero components. We propose a Lasso-type estimator $\hatβ^{Quad}$ (where '$Quad$' stands for quadratic) which is based on two penalty terms. The first one is the $\ell_1$ norm of the regression coefficients used to exploit the sparsity of the regression as done by the Lasso estimator, whereas the second is a quadratic penalty term introduced to capture some additional information on the setting of the problem. We detail two special cases: the Elastic-Net $\hatβ^{EN}$, which deals with sparse problems where correlations between variables may exist; and the Smooth-Lasso $\hatβ^{SL}$, which responds to sparse problems where successive regression coefficients are known to vary slowly (in some situations, this can also be interpreted in terms of correlations between successive variables). From a theoretical point of view, we establish variable selection consistency results and show that $\hatβ^{Quad}$ achieves a Sparsity Inequality, \textit i.e., a bound in terms of the number of non-zero components of the 'true' regression vector. These results are provided under a weaker assumption on the Gram matrix than the one used by the Lasso. In some situations this guarantees a significant improvement over the Lasso. Furthermore, a simulation study is conducted and shows that the S-Lasso $\hatβ^{SL}$ performs better than known methods as the Lasso, the Elastic-Net $\hatβ^{EN}$, and the Fused-Lasso with respect to the estimation accuracy. This is especially the case when the regression vector is 'smooth', \textit i.e., when the variations between successive coefficients of the unknown parameter of the regression are small. The study also reveals that the theoretical calibration of the tuning parameters and the one based on 10 fold cross validation imply two S-Lasso solutions with close performance. △ Less

Submitted 7 October, 2011; v1 submitted 25 March, 2010; originally announced March 2010.

On the conditions used to prove oracle results for the Lasso

Authors: Sara A. van de Geer, Peter Bühlmann

Abstract: Oracle inequalities and variable selection properties for the Lasso in linear models have been established under a variety of different assumptions on the design matrix. We show in this paper how the different conditions and concepts relate to each other. The restricted eigenvalue condition (Bickel et al., 2009) or the slightly weaker compatibility condition (van de Geer, 2007) are sufficient fo… ▽ More Oracle inequalities and variable selection properties for the Lasso in linear models have been established under a variety of different assumptions on the design matrix. We show in this paper how the different conditions and concepts relate to each other. The restricted eigenvalue condition (Bickel et al., 2009) or the slightly weaker compatibility condition (van de Geer, 2007) are sufficient for oracle results. We argue that both these conditions allow for a fairly general class of design matrices. Hence, optimality of the Lasso for prediction and estimation holds for more general situations than what it appears from coherence (Bunea et al, 2007b,c) or restricted isometry (Candes and Tao, 2005) assumptions. △ Less

Submitted 5 October, 2009; originally announced October 2009.

Comments: 33 pages, 1 figure

Journal ref: Electronic Journal of Statistics, 3, (2009), 1360-1392

High-dimensional generalized linear models and the lasso

Authors: Sara A. van de Geer

Abstract: We consider high-dimensional generalized linear models with Lipschitz loss functions, and prove a nonasymptotic oracle inequality for the empirical risk minimizer with Lasso penalty. The penalty is based on the coefficients in the linear predictor, after normalization with the empirical norm. The examples include logistic regression, density estimation and classification with hinge loss. Least s… ▽ More We consider high-dimensional generalized linear models with Lipschitz loss functions, and prove a nonasymptotic oracle inequality for the empirical risk minimizer with Lasso penalty. The penalty is based on the coefficients in the linear predictor, after normalization with the empirical norm. The examples include logistic regression, density estimation and classification with hinge loss. Least squares regression is also discussed. △ Less

Submitted 4 April, 2008; originally announced April 2008.

Comments: Published in at http://dx.doi.org/10.1214/009053607000000929 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-AOS-AOS0344 MSC Class: 62G08 (Primary)

Journal ref: Annals of Statistics 2008, Vol. 36, No. 2, 614-645

On non-asymptotic bounds for estimation in generalized linear models with highly correlated design

Authors: Sara A. van de Geer

Abstract: We study a high-dimensional generalized linear model and penalized empirical risk minimization with $\ell_1$ penalty. Our aim is to provide a non-trivial illustration that non-asymptotic bounds for the estimator can be obtained without relying on the chaining technique and/or the peeling device. We study a high-dimensional generalized linear model and penalized empirical risk minimization with $\ell_1$ penalty. Our aim is to provide a non-trivial illustration that non-asymptotic bounds for the estimator can be obtained without relying on the chaining technique and/or the peeling device. △ Less

Submitted 6 September, 2007; originally announced September 2007.

Comments: Published at http://dx.doi.org/10.1214/074921707000000319 in the IMS Lecture Notes Monograph Series (http://www.imstat.org/publications/lecnotes.htm) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-LNMS55-LNMS5507 MSC Class: 62G08 (Primary)

Journal ref: IMS Lecture Notes Monograph Series 2007, Vol. 55, 121-134

Square Root Penalty: Adaptation to the Margin in Classification and in Edge Estimation

Authors: A. B. Tsybakov, S. A. van de Geer

Abstract: We consider the problem of adaptation to the margin in binary classification. We suggest a penalized empirical risk minimization classifier that adaptively attains, up to a logarithmic factor, fast optimal rates of convergence for the excess risk, that is, rates that can be faster than n^{-1/2}, where n is the sample size. We show that our method also gives adaptive estimators for the problem of… ▽ More We consider the problem of adaptation to the margin in binary classification. We suggest a penalized empirical risk minimization classifier that adaptively attains, up to a logarithmic factor, fast optimal rates of convergence for the excess risk, that is, rates that can be faster than n^{-1/2}, where n is the sample size. We show that our method also gives adaptive estimators for the problem of edge estimation. △ Less

Submitted 21 July, 2005; originally announced July 2005.

Comments: Published at http://dx.doi.org/10.1214/009053604000001066 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-AOS-AOS031 MSC Class: 62G07; 62G08; 62H30; 68T10 (Primary)

Journal ref: Annals of Statistics 2005, Vol. 33, No. 3, 1203-1224