Showing 1–10 of 10 results for author: Leeb, H

A (tight) upper bound for the length of confidence intervals with conditional coverage

Authors: Danijel Kivaranovic, Hannes Leeb

Abstract: We show that two popular selective inference procedures, namely data carving (Fithian et al., 2017) and selection with a randomized response (Tian et al., 2018b), when combined with the polyhedral method (Lee et al., 2016), result in confidence intervals whose length is bounded. This contrasts results for confidence intervals based on the polyhedral method alone, whose expected length is typically… ▽ More We show that two popular selective inference procedures, namely data carving (Fithian et al., 2017) and selection with a randomized response (Tian et al., 2018b), when combined with the polyhedral method (Lee et al., 2016), result in confidence intervals whose length is bounded. This contrasts results for confidence intervals based on the polyhedral method alone, whose expected length is typically infinite (Kivaranovic and Leeb, 2020). Moreover, we show that these two procedures always dominate corresponding sample-splitting methods in terms of interval length. △ Less

Submitted 21 February, 2024; v1 submitted 24 July, 2020; originally announced July 2020.

Comments: 23 pages, 2 figures

Conformal prediction intervals for the individual treatment effect

Authors: Danijel Kivaranovic, Robin Ristl, Martin Posch, Hannes Leeb

Abstract: We propose several prediction intervals procedures for the individual treatment effect with either finite-sample or asymptotic coverage guarantee in a non-parametric regression setting, where non-linear regression functions, heteroskedasticity and non-Gaussianity are allowed. The construct the prediction intervals we use the conformal method of Vovk et al. (2005). In extensive simulations, we comp… ▽ More We propose several prediction intervals procedures for the individual treatment effect with either finite-sample or asymptotic coverage guarantee in a non-parametric regression setting, where non-linear regression functions, heteroskedasticity and non-Gaussianity are allowed. The construct the prediction intervals we use the conformal method of Vovk et al. (2005). In extensive simulations, we compare the coverage probability and interval length of our prediction interval procedures. We demonstrate that complex learning algorithms, such as neural networks, can lead to narrower prediction intervals than simple algorithms, such as linear regression, if the sample size is large enough. △ Less

Submitted 2 June, 2020; originally announced June 2020.

Comments: 32 pages, 2 figures

Adaptive, Distribution-Free Prediction Intervals for Deep Networks

Authors: Danijel Kivaranovic, Kory D. Johnson, Hannes Leeb

Abstract: The machine learning literature contains several constructions for prediction intervals that are intuitively reasonable but ultimately ad-hoc in that they do not come with provable performance guarantees. We present methods from the statistics literature that can be used efficiently with neural networks under minimal assumptions with guaranteed performance. We propose a neural network that outputs… ▽ More The machine learning literature contains several constructions for prediction intervals that are intuitively reasonable but ultimately ad-hoc in that they do not come with provable performance guarantees. We present methods from the statistics literature that can be used efficiently with neural networks under minimal assumptions with guaranteed performance. We propose a neural network that outputs three values instead of a single point estimate and optimizes a loss function motivated by the standard quantile regression loss. We provide two prediction interval methods with finite sample coverage guarantees solely under the assumption that the observations are independent and identically distributed. The first method leverages the conformal inference framework and provides average coverage. The second method provides a new, stronger guarantee by conditioning on the observed data. Lastly, our loss function does not compromise the predictive accuracy of the network like other prediction interval methods. We demonstrate the ease of use of our procedures as well as its improvements over other methods on both simulated and real data. As most deep networks can easily be modified by our method to output predictions with valid prediction intervals, its use should become standard practice, much like reporting standard errors along with mean estimates. △ Less

Submitted 24 February, 2020; v1 submitted 25 May, 2019; originally announced May 2019.

Discussion on "Model Confidence Bounds for Variable Selection" by Yang Li, Yuetian Luo, Davide Ferrari, Xiaonan Hu, and Yichen Qin

Authors: Hannes Leeb, Benedikt M. Pötscher, Danijel Kivaranovic

Abstract: This is a comment on the article mentioned in the title. This is a comment on the article mentioned in the title. △ Less

Submitted 20 December, 2018; v1 submitted 19 December, 2018; originally announced December 2018.

Journal ref: Biometrics 75 (2019), 407-410

Valid confidence intervals for post-model-selection predictors

Authors: François Bachoc, Hannes Leeb, Benedikt M. Pötscher

Abstract: We consider inference post-model-selection in linear regression. In this setting, Berk et al.(2013) recently introduced a class of confidence sets, the so-called PoSI intervals, that cover a certain non-standard quantity of interest with a user-specified minimal coverage probability, irrespective of the model selection procedure that is being used. In this paper, we generalize the PoSI intervals t… ▽ More We consider inference post-model-selection in linear regression. In this setting, Berk et al.(2013) recently introduced a class of confidence sets, the so-called PoSI intervals, that cover a certain non-standard quantity of interest with a user-specified minimal coverage probability, irrespective of the model selection procedure that is being used. In this paper, we generalize the PoSI intervals to post-model-selection predictors. △ Less

Submitted 27 January, 2017; v1 submitted 15 December, 2014; originally announced December 2014.

Comments: Some material added. Some restructuring of the paper. Some minor errors corrected

MSC Class: 62F25; 62J05

Journal ref: Annals of Statistics 47 (2019), 1475-1504

Evaluation and selection of models for out-of-sample prediction when the sample size is small relative to the complexity of the data-generating process

Authors: Hannes Leeb

Abstract: In regression with random design, we study the problem of selecting a model that performs well for out-of-sample prediction. We do not assume that any of the candidate models under consideration are correct. Our analysis is based on explicit finite-sample results. Our main findings differ from those of other analyses that are based on traditional large-sample limit approximations because we cons… ▽ More In regression with random design, we study the problem of selecting a model that performs well for out-of-sample prediction. We do not assume that any of the candidate models under consideration are correct. Our analysis is based on explicit finite-sample results. Our main findings differ from those of other analyses that are based on traditional large-sample limit approximations because we consider a situation where the sample size is small relative to the complexity of the data-generating process, in the sense that the number of parameters in a `good' model is of the same order as sample size. Also, we allow for the case where the number of candidate models is (much) larger than sample size. △ Less

Submitted 24 October, 2008; v1 submitted 22 February, 2008; originally announced February 2008.

Comments: Published in at http://dx.doi.org/10.3150/08-BEJ127 the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm)

Report number: IMS-BEJ-BEJ127

Journal ref: Bernoulli 2008, Vol. 14, No. 3, 661-690

On the Distribution of Penalized Maximum Likelihood Estimators: The LASSO, SCAD, and Thresholding

Authors: Benedikt M. Potscher, Hannes Leeb

Abstract: We study the distributions of the LASSO, SCAD, and thresholding estimators, in finite samples and in the large-sample limit. The asymptotic distributions are derived for both the case where the estimators are tuned to perform consistent model selection and for the case where the estimators are tuned to perform conservative model selection. Our findings complement those of Knight and Fu (2000) an… ▽ More We study the distributions of the LASSO, SCAD, and thresholding estimators, in finite samples and in the large-sample limit. The asymptotic distributions are derived for both the case where the estimators are tuned to perform consistent model selection and for the case where the estimators are tuned to perform conservative model selection. Our findings complement those of Knight and Fu (2000) and Fan and Li (2001). We show that the distributions are typically highly nonnormal regardless of how the estimator is tuned, and that this property persists in large samples. The uniform convergence rate of these estimators is also obtained, and is shown to be slower than 1/root(n) in case the estimator is tuned to perform consistent model selection. An impossibility result regarding estimation of the estimators' distribution function is also provided. △ Less

Submitted 17 April, 2009; v1 submitted 5 November, 2007; originally announced November 2007.

MSC Class: 62J07; 62F12; 62E15.

Journal ref: J. Multivariate Anal. 100 (2009) 2065-2082

Can One Estimate The Unconditional Distribution of Post-Model-Selection Estimators?

Authors: Hannes Leeb, Benedikt M. Poetscher

Abstract: We consider the problem of estimating the unconditional distribution of a post-model-selection estimator. The notion of a post-model-selection estimator here refers to the combined procedure resulting from first selecting a model (e.g., by a model selection criterion like AIC or by a hypothesis testing procedure) and then estimating the parameters in the selected model (e.g., by least-squares or… ▽ More We consider the problem of estimating the unconditional distribution of a post-model-selection estimator. The notion of a post-model-selection estimator here refers to the combined procedure resulting from first selecting a model (e.g., by a model selection criterion like AIC or by a hypothesis testing procedure) and then estimating the parameters in the selected model (e.g., by least-squares or maximum likelihood), all based on the same data set. We show that it is impossible to estimate the unconditional distribution with reasonable accuracy even asymptotically. In particular, we show that no estimator for this distribution can be uniformly consistent (not even locally). This follows as a corollary to (local) minimax lower bounds on the performance of estimators for the distribution; performance is here measured by the probability that the estimation error exceeds a given threshold. These lower bounds are shown to approach 1/2 or even 1 in large samples, depending on the situation considered. Similar impossibility results are also obtained for the distribution of linear functions (e.g., predictors) of the post-model-selection estimator. △ Less

Submitted 12 April, 2007; originally announced April 2007.

MSC Class: 62F10; 62F12; 62J05; 62J07; 62C05

Journal ref: Econometric Theory 24 (2008)

Sparse Estimators and the Oracle Property, or the Return of Hodges' Estimator

Authors: Hannes Leeb, Benedikt M. Poetscher

Abstract: We point out some pitfalls related to the concept of an oracle property as used in Fan and Li (2001, 2002, 2004) which are reminiscent of the well-known pitfalls related to Hodges' estimator. The oracle property is often a consequence of sparsity of an estimator. We show that any estimator satisfying a sparsity property has maximal risk that converges to the supremum of the loss function; in par… ▽ More We point out some pitfalls related to the concept of an oracle property as used in Fan and Li (2001, 2002, 2004) which are reminiscent of the well-known pitfalls related to Hodges' estimator. The oracle property is often a consequence of sparsity of an estimator. We show that any estimator satisfying a sparsity property has maximal risk that converges to the supremum of the loss function; in particular, the maximal risk diverges to infinity whenever the loss function is unbounded. For ease of presentation the result is set in the framework of a linear regression model, but generalizes far beyond that setting. In a Monte Carlo study we also assess the extent of the problem in finite samples for the smoothly clipped absolute deviation (SCAD) estimator introduced in Fan and Li (2001). We find that this estimator can perform rather poorly in finite samples and that its worst-case performance relative to maximum likelihood deteriorates with increasing sample size when the estimator is tuned to sparsity. △ Less

Submitted 11 April, 2007; originally announced April 2007.

Comments: 18 pages, 5 figures

MSC Class: 62J07; 62C99; 62E20; 62F10; 62F12

Journal ref: Journal of Econometrics (2008)

Can one estimate the conditional distribution of post-model-selection estimators?

Authors: Hannes Leeb, Benedikt M. Pötscher

Abstract: We consider the problem of estimating the conditional distribution of a post-model-selection estimator where the conditioning is on the selected model. The notion of a post-model-selection estimator here refers to the combined procedure resulting from first selecting a model (e.g., by a model selection criterion such as AIC or by a hypothesis testing procedure) and then estimating the parameters… ▽ More We consider the problem of estimating the conditional distribution of a post-model-selection estimator where the conditioning is on the selected model. The notion of a post-model-selection estimator here refers to the combined procedure resulting from first selecting a model (e.g., by a model selection criterion such as AIC or by a hypothesis testing procedure) and then estimating the parameters in the selected model (e.g., by least-squares or maximum likelihood), all based on the same data set. We show that it is impossible to estimate this distribution with reasonable accuracy even asymptotically. In particular, we show that no estimator for this distribution can be uniformly consistent (not even locally). This follows as a corollary to (local) minimax lower bounds on the performance of estimators for this distribution. Similar impossibility results are also obtained for the conditional distribution of linear functions (e.g., predictors) of the post-model-selection estimator. △ Less

Submitted 23 February, 2007; originally announced February 2007.

Comments: Published at http://dx.doi.org/10.1214/009053606000000821 in the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-AOS-AOS0185 MSC Class: 62F10; 62F12; 62J05; 62J07; 62C05 (Primary)

Journal ref: Annals of Statistics 2006, Vol. 34, No. 5, 2554-2591