-
Adapting to Misspecification
Authors:
Timothy B. Armstrong,
Patrick Kline,
Liyang Sun
Abstract:
Empirical research typically involves a robustness-efficiency tradeoff. A researcher seeking to estimate a scalar parameter can invoke strong assumptions to motivate a restricted estimator that is precise but may be heavily biased, or they can relax some of these assumptions to motivate a more robust, but variable, unrestricted estimator. When a bound on the bias of the restricted estimator is ava…
▽ More
Empirical research typically involves a robustness-efficiency tradeoff. A researcher seeking to estimate a scalar parameter can invoke strong assumptions to motivate a restricted estimator that is precise but may be heavily biased, or they can relax some of these assumptions to motivate a more robust, but variable, unrestricted estimator. When a bound on the bias of the restricted estimator is available, it is optimal to shrink the unrestricted estimator towards the restricted estimator. For settings where a bound on the bias of the restricted estimator is unknown, we propose adaptive estimators that minimize the percentage increase in worst case risk relative to an oracle that knows the bound. We show that adaptive estimators solve a weighted convex minimax problem and provide lookup tables facilitating their rapid computation. Revisiting some well known empirical studies where questions of model specification arise, we examine the advantages of adapting to -- rather than testing for -- misspecification.
△ Less
Submitted 27 August, 2024; v1 submitted 23 May, 2023;
originally announced May 2023.
-
Robust Estimation and Inference in Panels with Interactive Fixed Effects
Authors:
Timothy B. Armstrong,
Martin Weidner,
Andrei Zeleneev
Abstract:
We consider estimation and inference for a regression coefficient in panels with interactive fixed effects (i.e., with a factor structure). We demonstrate that existing estimators and confidence intervals (CIs) can be heavily biased and size-distorted when some of the factors are weak. We propose estimators with improved rates of convergence and bias-aware CIs that remain valid uniformly, regardle…
▽ More
We consider estimation and inference for a regression coefficient in panels with interactive fixed effects (i.e., with a factor structure). We demonstrate that existing estimators and confidence intervals (CIs) can be heavily biased and size-distorted when some of the factors are weak. We propose estimators with improved rates of convergence and bias-aware CIs that remain valid uniformly, regardless of factor strength. Our approach applies the theory of minimax linear estimation to form a debiased estimate, using a nuclear norm bound on the error of an initial estimate of the interactive fixed effects. Our resulting bias-aware CIs take into account the remaining bias caused by weak factors. Monte Carlo experiments show substantial improvements over conventional methods when factors are weak, with minimal costs to estimation accuracy when factors are strong.
△ Less
Submitted 11 May, 2025; v1 submitted 12 October, 2022;
originally announced October 2022.
-
False Discovery Rate Adjustments for Average Significance Level Controlling Tests
Authors:
Timothy B. Armstrong
Abstract:
Multiple testing adjustments, such as the Benjamini and Hochberg (1995) step-up procedure for controlling the false discovery rate (FDR), are typically applied to families of tests that control significance level in the classical sense: for each individual test, the probability of false rejection is no greater than the nominal level. In this paper, we consider tests that satisfy only a weaker noti…
▽ More
Multiple testing adjustments, such as the Benjamini and Hochberg (1995) step-up procedure for controlling the false discovery rate (FDR), are typically applied to families of tests that control significance level in the classical sense: for each individual test, the probability of false rejection is no greater than the nominal level. In this paper, we consider tests that satisfy only a weaker notion of significance level control, in which the probability of false rejection need only be controlled on average over the hypotheses. We find that the Benjamini and Hochberg (1995) step-up procedure still controls FDR in the asymptotic regime with many weakly dependent $p$-values, and that certain adjustments for dependent $p$-values such as the Benjamini and Yekutieli (2001) procedure continue to yield FDR control in finite samples. Our results open the door to FDR controlling procedures in nonparametric and high dimensional settings where weakening the notion of inference allows for large power improvements.
△ Less
Submitted 15 May, 2025; v1 submitted 27 September, 2022;
originally announced September 2022.
-
Asymptotic Efficiency Bounds for a Class of Experimental Designs
Authors:
Timothy B. Armstrong
Abstract:
We consider an experimental design setting in which units are assigned to treatment after being sampled sequentially from an infinite population. We derive asymptotic efficiency bounds that apply to data from any experiment that assigns treatment as a (possibly randomized) function of covariates and past outcome data, including stratification on covariates and adaptive designs. For estimating the…
▽ More
We consider an experimental design setting in which units are assigned to treatment after being sampled sequentially from an infinite population. We derive asymptotic efficiency bounds that apply to data from any experiment that assigns treatment as a (possibly randomized) function of covariates and past outcome data, including stratification on covariates and adaptive designs. For estimating the average treatment effect of a binary treatment, our results show that no further first order asymptotic efficiency improvement is possible relative to an estimator that achieves the Hahn (1998) bound in an experimental design where the propensity score is chosen to minimize this bound. Our results also apply to settings with multiple treatments with possible constraints on treatment, as well as covariate based sampling of a single outcome.
△ Less
Submitted 13 May, 2025; v1 submitted 5 May, 2022;
originally announced May 2022.
-
Bias-Aware Inference in Regularized Regression Models
Authors:
Timothy B. Armstrong,
Michal Kolesár,
Soonwoo Kwon
Abstract:
We consider inference on a scalar regression coefficient under a constraint on the magnitude of the control coefficients. A class of estimators based on a regularized propensity score regression is shown to exactly solve a tradeoff between worst-case bias and variance. We derive confidence intervals (CIs) based on these estimators that are bias-aware: they account for the possible bias of the esti…
▽ More
We consider inference on a scalar regression coefficient under a constraint on the magnitude of the control coefficients. A class of estimators based on a regularized propensity score regression is shown to exactly solve a tradeoff between worst-case bias and variance. We derive confidence intervals (CIs) based on these estimators that are bias-aware: they account for the possible bias of the estimator. Under homoskedastic Gaussian errors, these estimators and CIs are near-optimal in finite samples for MSE and CI length. We also provide conditions for asymptotic validity of the CI with unknown and possibly heteroskedastic error distribution, and derive novel optimal rates of convergence under high-dimensional asymptotics that allow the number of regressors to increase more quickly than the number of observations. Extensive simulations and an empirical application illustrate the performance of our methods.
△ Less
Submitted 10 August, 2023; v1 submitted 29 December, 2020;
originally announced December 2020.
-
Robust Empirical Bayes Confidence Intervals
Authors:
Timothy B. Armstrong,
Michal Kolesár,
Mikkel Plagborg-Møller
Abstract:
We construct robust empirical Bayes confidence intervals (EBCIs) in a normal means problem. The intervals are centered at the usual linear empirical Bayes estimator, but use a critical value accounting for shrinkage. Parametric EBCIs that assume a normal distribution for the means (Morris, 1983b) may substantially undercover when this assumption is violated. In contrast, our EBCIs control coverage…
▽ More
We construct robust empirical Bayes confidence intervals (EBCIs) in a normal means problem. The intervals are centered at the usual linear empirical Bayes estimator, but use a critical value accounting for shrinkage. Parametric EBCIs that assume a normal distribution for the means (Morris, 1983b) may substantially undercover when this assumption is violated. In contrast, our EBCIs control coverage regardless of the means distribution, while remaining close in length to the parametric EBCIs when the means are indeed Gaussian. If the means are treated as fixed, our EBCIs have an average coverage guarantee: the coverage probability is at least $1 - α$ on average across the $n$ EBCIs for each of the means. Our empirical application considers the effects of U.S. neighborhoods on intergenerational mobility.
△ Less
Submitted 14 May, 2022; v1 submitted 7 April, 2020;
originally announced April 2020.
-
Sensitivity Analysis using Approximate Moment Condition Models
Authors:
Timothy B. Armstrong,
Michal Kolesár
Abstract:
We consider inference in models defined by approximate moment conditions. We show that near-optimal confidence intervals (CIs) can be formed by taking a generalized method of moments (GMM) estimator, and adding and subtracting the standard error times a critical value that takes into account the potential bias from misspecification of the moment conditions. In order to optimize performance under p…
▽ More
We consider inference in models defined by approximate moment conditions. We show that near-optimal confidence intervals (CIs) can be formed by taking a generalized method of moments (GMM) estimator, and adding and subtracting the standard error times a critical value that takes into account the potential bias from misspecification of the moment conditions. In order to optimize performance under potential misspecification, the weighting matrix for this GMM estimator takes into account this potential bias, and therefore differs from the one that is optimal under correct specification. To formally show the near-optimality of these CIs, we develop asymptotic efficiency bounds for inference in the locally misspecified GMM setting. These bounds may be of independent interest, due to their implications for the possibility of using moment selection procedures when conducting inference in moment condition models. We apply our methods in an empirical application to automobile demand, and show that adjusting the weighting matrix can shrink the CIs by a factor of 3 or more.
△ Less
Submitted 29 July, 2020; v1 submitted 22 August, 2018;
originally announced August 2018.
-
Finite-Sample Optimal Estimation and Inference on Average Treatment Effects Under Unconfoundedness
Authors:
Timothy B. Armstrong,
Michal Kolesár
Abstract:
We consider estimation and inference on average treatment effects under unconfoundedness conditional on the realizations of the treatment variable and covariates. Given nonparametric smoothness and/or shape restrictions on the conditional mean of the outcome variable, we derive estimators and confidence intervals (CIs) that are optimal in finite samples when the regression errors are normal with k…
▽ More
We consider estimation and inference on average treatment effects under unconfoundedness conditional on the realizations of the treatment variable and covariates. Given nonparametric smoothness and/or shape restrictions on the conditional mean of the outcome variable, we derive estimators and confidence intervals (CIs) that are optimal in finite samples when the regression errors are normal with known variance. In contrast to conventional CIs, our CIs use a larger critical value that explicitly takes into account the potential bias of the estimator. When the error distribution is unknown, feasible versions of our CIs are valid asymptotically, even when $\sqrt{n}$-inference is not possible due to lack of overlap, or low smoothness of the conditional mean. We also derive the minimum smoothness conditions on the conditional mean that are necessary for $\sqrt{n}$-inference. When the conditional mean is restricted to be Lipschitz with a large enough bound on the Lipschitz constant, the optimal estimator reduces to a matching estimator with the number of matches set to one. We illustrate our methods in an application to the National Supported Work Demonstration.
△ Less
Submitted 18 January, 2021; v1 submitted 12 December, 2017;
originally announced December 2017.
-
Simple and Honest Confidence Intervals in Nonparametric Regression
Authors:
Timothy B. Armstrong,
Michal Kolesár
Abstract:
We consider the problem of constructing honest confidence intervals (CIs) for a scalar parameter of interest, such as the regression discontinuity parameter, in nonparametric regression based on kernel or local polynomial estimators. To ensure that our CIs are honest, we use critical values that take into account the possible bias of the estimator upon which the CIs are based. We show that this ap…
▽ More
We consider the problem of constructing honest confidence intervals (CIs) for a scalar parameter of interest, such as the regression discontinuity parameter, in nonparametric regression based on kernel or local polynomial estimators. To ensure that our CIs are honest, we use critical values that take into account the possible bias of the estimator upon which the CIs are based. We show that this approach leads to CIs that are more efficient than conventional CIs that achieve coverage by undersmoothing or subtracting an estimate of the bias. We give sharp efficiency bounds of using different kernels, and derive the optimal bandwidth for constructing honest CIs. We show that using the bandwidth that minimizes the maximum mean-squared error results in CIs that are nearly efficient and that in this case, the critical value depends only on the rate of convergence. For the common case in which the rate of convergence is $n^{-2/5}$, the appropriate critical value for 95% CIs is 2.18, rather than the usual 1.96 critical value. We illustrate our results in a Monte Carlo analysis and an empirical application.
△ Less
Submitted 28 August, 2019; v1 submitted 3 June, 2016;
originally announced June 2016.
-
Optimal inference in a class of regression models
Authors:
Timothy B. Armstrong,
Michal Kolesár
Abstract:
We consider the problem of constructing confidence intervals (CIs) for a linear functional of a regression function, such as its value at a point, the regression discontinuity parameter, or a regression coefficient in a linear or partly linear regression. Our main assumption is that the regression function is known to lie in a convex function class, which covers most smoothness and/or shape assump…
▽ More
We consider the problem of constructing confidence intervals (CIs) for a linear functional of a regression function, such as its value at a point, the regression discontinuity parameter, or a regression coefficient in a linear or partly linear regression. Our main assumption is that the regression function is known to lie in a convex function class, which covers most smoothness and/or shape assumptions used in econometrics. We derive finite-sample optimal CIs and sharp efficiency bounds under normal errors with known variance. We show that these results translate to uniform (over the function class) asymptotic results when the error distribution is not known. When the function class is centrosymmetric, these efficiency bounds imply that minimax CIs are close to efficient at smooth regression functions. This implies, in particular, that it is impossible to form CIs that are tighter using data-dependent tuning parameters, and maintain coverage over the whole function class. We specialize our results to inference on the regression discontinuity parameter, and illustrate them in simulations and an empirical application.
△ Less
Submitted 22 November, 2017; v1 submitted 18 November, 2015;
originally announced November 2015.
-
Unbiased Instrumental Variables Estimation Under Known First-Stage Sign
Authors:
Isaiah Andrews,
Timothy B. Armstrong
Abstract:
We derive mean-unbiased estimators for the structural parameter in instrumental variables models with a single endogenous regressor where the sign of one or more first stage coefficients is known. In the case with a single instrument, there is a unique non-randomized unbiased estimator based on the reduced-form and first-stage regression estimates. For cases with multiple instruments we propose a…
▽ More
We derive mean-unbiased estimators for the structural parameter in instrumental variables models with a single endogenous regressor where the sign of one or more first stage coefficients is known. In the case with a single instrument, there is a unique non-randomized unbiased estimator based on the reduced-form and first-stage regression estimates. For cases with multiple instruments we propose a class of unbiased estimators and show that an estimator within this class is efficient when the instruments are strong. We show numerically that unbiasedness does not come at a cost of increased dispersion in models with a single instrument: in this case the unbiased estimator is less dispersed than the 2SLS estimator. Our finite-sample results apply to normal models with known variance for the reduced-form errors, and imply analogous results under weak instrument asymptotics with an unknown error distribution.
△ Less
Submitted 2 December, 2016; v1 submitted 26 January, 2015;
originally announced January 2015.
-
A Note on Minimax Testing and Confidence Intervals in Moment Inequality Models
Authors:
Timothy B. Armstrong
Abstract:
This note uses a simple example to show how moment inequality models used in the empirical economics literature lead to general minimax relative efficiency comparisons. The main point is that such models involve inference on a low dimensional parameter, which leads naturally to a definition of "distance" that, in full generality, would be arbitrary in minimax testing problems. This definition of d…
▽ More
This note uses a simple example to show how moment inequality models used in the empirical economics literature lead to general minimax relative efficiency comparisons. The main point is that such models involve inference on a low dimensional parameter, which leads naturally to a definition of "distance" that, in full generality, would be arbitrary in minimax testing problems. This definition of distance is justified by the fact that it leads to a duality between minimaxity of confidence intervals and tests, which does not hold for other definitions of distance. Thus, the use of moment inequalities for inference in a low dimensional parametric model places additional structure on the testing problem, which leads to stronger conclusions regarding minimax relative efficiency than would otherwise be possible.
△ Less
Submitted 17 December, 2014;
originally announced December 2014.
-
A Simple Adjustment for Bandwidth Snooping
Authors:
Timothy B. Armstrong,
Michal Kolesár
Abstract:
Kernel-based estimators such as local polynomial estimators in regression discontinuity designs are often evaluated at multiple bandwidths as a form of sensitivity analysis. However, if in the reported results, a researcher selects the bandwidth based on this analysis, the associated confidence intervals may not have correct coverage, even if the estimator is unbiased. This paper proposes a simple…
▽ More
Kernel-based estimators such as local polynomial estimators in regression discontinuity designs are often evaluated at multiple bandwidths as a form of sensitivity analysis. However, if in the reported results, a researcher selects the bandwidth based on this analysis, the associated confidence intervals may not have correct coverage, even if the estimator is unbiased. This paper proposes a simple adjustment that gives correct coverage in such situations: replace the normal quantile with a critical value that depends only on the kernel and ratio of the maximum and minimum bandwidths the researcher has entertained. We tabulate these critical values and quantify the loss in coverage for conventional confidence intervals. For a range of relevant cases, a conventional 95% confidence interval has coverage between 70% and 90%, and our adjustment amounts to replacing the conventional critical value 1.96 with a number between 2.2 and 2.8. Our results also apply to other settings involving trimmed data, such as trimming to ensure overlap in treatment effect estimation. We illustrate our approach with three empirical applications.
△ Less
Submitted 28 June, 2017; v1 submitted 30 November, 2014;
originally announced December 2014.
-
On the Choice of Test Statistic for Conditional Moment Inequalities
Authors:
Timothy B. Armstrong
Abstract:
This paper derives asymptotic approximations to the power of Cramer-von Mises (CvM) style tests for inference on a finite dimensional parameter defined by conditional moment inequalities in the case where the parameter is set identified. Combined with power results for Kolmogorov-Smirnov (KS) tests, these results can be used to choose the optimal test statistic, weighting function and, for tests b…
▽ More
This paper derives asymptotic approximations to the power of Cramer-von Mises (CvM) style tests for inference on a finite dimensional parameter defined by conditional moment inequalities in the case where the parameter is set identified. Combined with power results for Kolmogorov-Smirnov (KS) tests, these results can be used to choose the optimal test statistic, weighting function and, for tests based on kernel estimates, kernel bandwidth. The results show that, in the setting considered here, KS tests are preferred to CvM tests, and that a truncated variance weighting is preferred to bounded weightings.
△ Less
Submitted 7 July, 2017; v1 submitted 17 October, 2014;
originally announced October 2014.
-
Multiscale Adaptive Inference on Conditional Moment Inequalities
Authors:
Timothy B. Armstrong,
Hock Peng Chan
Abstract:
This paper considers inference for conditional moment inequality models using a multiscale statistic. We derive the asymptotic distribution of this test statistic and use the result to propose feasible critical values that have a simple analytic formula, and to prove the asymptotic validity of a modified bootstrap procedure. The asymptotic distribution is extreme value, and the proof uses new tech…
▽ More
This paper considers inference for conditional moment inequality models using a multiscale statistic. We derive the asymptotic distribution of this test statistic and use the result to propose feasible critical values that have a simple analytic formula, and to prove the asymptotic validity of a modified bootstrap procedure. The asymptotic distribution is extreme value, and the proof uses new techniques to overcome several technical obstacles. The test detects local alternatives that approach the identified set at the best rate among available tests in a broad class of models, and is adaptive to the smoothness properties of the data generating process. Our results also have implications for the use of moment selection procedures in this setting. We provide a monte carlo study and an empirical illustration to inference in a regression model with endogenously censored and missing data.
△ Less
Submitted 8 December, 2015; v1 submitted 22 December, 2012;
originally announced December 2012.
-
On the Asymptotic Distribution of Variance Weighted KS Statistics
Authors:
Timothy B. Armstrong
Abstract:
This paper derives the asymptotic distribution of variance weighted Kolmogorov-Smirnov statistics for conditional moment inequality models for the case of a one dimensional covariate. The asymptotic distribution depends on the data generating process only through the variance of a single random variable, leading to critical values that can be calculated analytically. By arguments in Armstrong (201…
▽ More
This paper derives the asymptotic distribution of variance weighted Kolmogorov-Smirnov statistics for conditional moment inequality models for the case of a one dimensional covariate. The asymptotic distribution depends on the data generating process only through the variance of a single random variable, leading to critical values that can be calculated analytically. By arguments in Armstrong (2011b), the resulting tests achieve the best minimax rate for local alternatives out of available approaches in a broad class of settings.
△ Less
Submitted 1 February, 2012;
originally announced February 2012.
-
Asymptotically Exact Inference in Conditional Moment Inequality Models
Authors:
Timothy B. Armstrong
Abstract:
This paper derives the rate of convergence and asymptotic distribution for a class of Kolmogorov-Smirnov style test statistics for conditional moment inequality models for parameters on the boundary of the identified set under general conditions. In contrast to other moment inequality settings, the rate of convergence is faster than root-$n$, and the asymptotic distribution depends entirely on non…
▽ More
This paper derives the rate of convergence and asymptotic distribution for a class of Kolmogorov-Smirnov style test statistics for conditional moment inequality models for parameters on the boundary of the identified set under general conditions. In contrast to other moment inequality settings, the rate of convergence is faster than root-$n$, and the asymptotic distribution depends entirely on nonbinding moments. The results require the development of new techniques that draw a connection between moment selection, irregular identification, bandwidth selection and nonstandard M-estimation. Using these results, I propose tests that are more powerful than existing approaches for choosing critical values for this test statistic. I quantify the power improvement by showing that the new tests can detect alternatives that converge to points on the identified set at a faster rate than those detected by existing approaches. A monte carlo study confirms that the tests and the asymptotic approximations they use perform well in finite samples. In an application to a regression of prescription drug expenditures on income with interval data from the Health and Retirement Study, confidence regions based on the new tests are substantially tighter than those based on existing methods.
△ Less
Submitted 5 December, 2011;
originally announced December 2011.
-
Weighted KS Statistics for Inference on Conditional Moment Inequalities
Authors:
Timothy B. Armstrong
Abstract:
This paper proposes confidence regions for the identified set in conditional moment inequality models using Kolmogorov-Smirnov statistics with a truncated inverse variance weighting with increasing truncation points. The new weighting differs from those proposed in the literature in two important ways. First, confidence regions based on KS tests with the weighting function I propose converge to th…
▽ More
This paper proposes confidence regions for the identified set in conditional moment inequality models using Kolmogorov-Smirnov statistics with a truncated inverse variance weighting with increasing truncation points. The new weighting differs from those proposed in the literature in two important ways. First, confidence regions based on KS tests with the weighting function I propose converge to the identified set at a faster rate than existing procedures based on bounded weight functions in a broad class of models. This provides a theoretical justification for inverse variance weighting in this context, and contrasts with analogous results for conditional moment equalities in which optimal weighting only affects the asymptotic variance. Second, the new weighting changes the asymptotic behavior, including the rate of convergence, of the KS statistic itself, requiring a new asymptotic theory in choosing the critical value, which I provide. To make these comparisons, I derive rates of convergence for the confidence regions I propose along with new results for rates of convergence of existing estimators under a general set of conditions. A series of examples illustrates the broad applicability of the conditions. A monte carlo study examines the finite sample behavior of the confidence regions.
△ Less
Submitted 5 December, 2011;
originally announced December 2011.
