Showing 1–3 of 3 results for author: Hesterberg, T

Power Analysis for Experiments with Clustered Data, Ratio Metrics, and Regression for Covariate Adjustment

Authors: Tim Hesterberg, Ben Knight

Abstract: We describe how to calculate standard errors for A/B tests that include clustered data, ratio metrics, and/or covariate adjustment. We may do this for power analysis/sample size calculations prior to running an experiment using historical data, or after an experiment for hypothesis testing and confidence intervals. The different applications have a common framework, using the sample variance of ce… ▽ More We describe how to calculate standard errors for A/B tests that include clustered data, ratio metrics, and/or covariate adjustment. We may do this for power analysis/sample size calculations prior to running an experiment using historical data, or after an experiment for hypothesis testing and confidence intervals. The different applications have a common framework, using the sample variance of certain residuals. The framework is compatible with modular software, can be plugged into standard tools, doesn't require computing covariance matrices, and is numerically stable. Using this approach we estimate that covariate adjustment gives a median 66% variance reduction for a key metric, reducing experiment run time by 66%. △ Less

Submitted 10 June, 2024; originally announced June 2024.

Comments: 17 pages, 6 figures

MSC Class: Primary 62K99; Secondary 62P30 ACM Class: G.3

What Teachers Should Know about the Bootstrap: Resampling in the Undergraduate Statistics Curriculum

Authors: Tim Hesterberg

Abstract: I have three goals in this article: (1) To show the enormous potential of bootstrapping and permutation tests to help students understand statistical concepts including sampling distributions, standard errors, bias, confidence intervals, null distributions, and P-values. (2) To dig deeper, understand why these methods work and when they don't, things to watch out for, and how to deal with these is… ▽ More I have three goals in this article: (1) To show the enormous potential of bootstrapping and permutation tests to help students understand statistical concepts including sampling distributions, standard errors, bias, confidence intervals, null distributions, and P-values. (2) To dig deeper, understand why these methods work and when they don't, things to watch out for, and how to deal with these issues when teaching. (3) To change statistical practice---by comparing these methods to common t tests and intervals, we see how inaccurate the latter are; we confirm this with asymptotics. n >= 30 isn't enough---think n >= 5000. Resampling provides diagnostics, and more accurate alternatives. Sadly, the common bootstrap percentile interval badly under-covers in small samples; there are better alternatives. The tone is informal, with a few stories and jokes. △ Less

Submitted 19 November, 2014; originally announced November 2014.

Comments: 83 pages, 23 figures

MSC Class: 62-01 (Primary) 62G09 (Secondary) ACM Class: G.3; I.6.8; K.3.0

Least angle and $\ell_1$ penalized regression: A review

Authors: Tim Hesterberg, Nam Hee Choi, Lukas Meier, Chris Fraley

Abstract: Least Angle Regression is a promising technique for variable selection applications, offering a nice alternative to stepwise regression. It provides an explanation for the similar behavior of LASSO ($\ell_1$-penalized regression) and forward stagewise regression, and provides a fast implementation of both. The idea has caught on rapidly, and sparked a great deal of research interest. In this pap… ▽ More Least Angle Regression is a promising technique for variable selection applications, offering a nice alternative to stepwise regression. It provides an explanation for the similar behavior of LASSO ($\ell_1$-penalized regression) and forward stagewise regression, and provides a fast implementation of both. The idea has caught on rapidly, and sparked a great deal of research interest. In this paper, we give an overview of Least Angle Regression and the current state of related research. △ Less

Submitted 21 May, 2008; v1 submitted 7 February, 2008; originally announced February 2008.

Comments: Published in at http://dx.doi.org/10.1214/08-SS035 the Statistics Surveys (http://www.i-journals.org/ss/) by the Institute of Mathematical Statistics (http://www.imstat.org)

Report number: IMS-SS-SS_2008_35 MSC Class: 62J07 (Primary) 69J99 (Secondary)

Journal ref: Statistics Surveys 2008, Vol. 2, 61-93