-
An Axiomatic Approach to Comparing Sensitivity Parameters
Authors:
Paul Diegert,
Matthew A. Masten,
Alexandre Poirier
Abstract:
Many methods are available for assessing the importance of omitted variables. These methods typically make different, non-falsifiable assumptions. Hence the data alone cannot tell us which method is most appropriate. Since it is unreasonable to expect results to be robust against all possible robustness checks, researchers often use methods deemed "interpretable", a subjective criterion with no fo…
▽ More
Many methods are available for assessing the importance of omitted variables. These methods typically make different, non-falsifiable assumptions. Hence the data alone cannot tell us which method is most appropriate. Since it is unreasonable to expect results to be robust against all possible robustness checks, researchers often use methods deemed "interpretable", a subjective criterion with no formal definition. In contrast, we develop the first formal, axiomatic framework for comparing and selecting among these methods. Our framework is analogous to the standard approach for comparing estimators based on their sampling distributions. We propose that sensitivity parameters be selected based on their covariate sampling distributions, a design distribution of parameter values induced by an assumption on how covariates are assigned to be observed or unobserved. Using this idea, we define a new concept of parameter consistency, and argue that a reasonable sensitivity parameter should be consistent. We prove that the literature's most popular approach is inconsistent, while several alternatives are consistent.
△ Less
Submitted 20 June, 2025; v1 submitted 29 April, 2025;
originally announced April 2025.
-
A General Approach to Relaxing Unconfoundedness
Authors:
Matthew A. Masten,
Alexandre Poirier,
Muyang Ren
Abstract:
This paper defines a general class of relaxations of the unconfoundedness assumption. This class includes several previous approaches as special cases, including the marginal sensitivity model of Tan (2006). This class therefore allows us to precisely compare and contrast these previously disparate relaxations. We use this class to derive a variety of new identification results which can be used t…
▽ More
This paper defines a general class of relaxations of the unconfoundedness assumption. This class includes several previous approaches as special cases, including the marginal sensitivity model of Tan (2006). This class therefore allows us to precisely compare and contrast these previously disparate relaxations. We use this class to derive a variety of new identification results which can be used to assess sensitivity to unconfoundedness. In particular, the prior literature focuses on average parameters, like the average treatment effect (ATE). We move beyond averages by providing sharp bounds for a large class of parameters, including both the quantile treatment effect (QTE) and the distribution of treatment effects (DTE), results which were previously unknown even for the marginal sensitivity model.
△ Less
Submitted 25 January, 2025;
originally announced January 2025.
-
Quantifying the Internal Validity of Weighted Estimands
Authors:
Alexandre Poirier,
Tymon Słoczyński
Abstract:
In this paper we study a class of weighted estimands, which we define as parameters that can be expressed as weighted averages of the underlying heterogeneous treatment effects. The popular ordinary least squares (OLS), two-stage least squares (2SLS), and two-way fixed effects (TWFE) estimands are all special cases within our framework. Our focus is on answering two questions concerning weighted e…
▽ More
In this paper we study a class of weighted estimands, which we define as parameters that can be expressed as weighted averages of the underlying heterogeneous treatment effects. The popular ordinary least squares (OLS), two-stage least squares (2SLS), and two-way fixed effects (TWFE) estimands are all special cases within our framework. Our focus is on answering two questions concerning weighted estimands. First, under what conditions can they be interpreted as the average treatment effect for some (possibly latent) subpopulation? Second, when these conditions are satisfied, what is the upper bound on the size of that subpopulation, either in absolute terms or relative to a target population of interest? We argue that this upper bound provides a valuable diagnostic for empirical research. When a given weighted estimand corresponds to the average treatment effect for a small subset of the population of interest, we say its internal validity is low. Our paper develops practical tools to quantify the internal validity of weighted estimands.
△ Less
Submitted 17 May, 2025; v1 submitted 22 April, 2024;
originally announced April 2024.
-
The Effect of Omitted Variables on the Sign of Regression Coefficients
Authors:
Matthew A. Masten,
Alexandre Poirier
Abstract:
We show that, depending on how the impact of omitted variables is measured, it can be substantially easier for omitted variables to flip coefficient signs than to drive them to zero. This behavior occurs with "Oster's delta" (Oster 2019), a widely reported robustness measure. Consequently, any time this measure is large -- suggesting that omitted variables may be unimportant -- a much smaller valu…
▽ More
We show that, depending on how the impact of omitted variables is measured, it can be substantially easier for omitted variables to flip coefficient signs than to drive them to zero. This behavior occurs with "Oster's delta" (Oster 2019), a widely reported robustness measure. Consequently, any time this measure is large -- suggesting that omitted variables may be unimportant -- a much smaller value reverses the sign of the parameter of interest. We propose a modified measure of robustness to address this concern. We illustrate our results in four empirical applications and two meta-analyses. We implement our methods in the companion Stata module regsensitivity.
△ Less
Submitted 29 June, 2025; v1 submitted 31 July, 2022;
originally announced August 2022.
-
Assessing Omitted Variable Bias when the Controls are Endogenous
Authors:
Paul Diegert,
Matthew A. Masten,
Alexandre Poirier
Abstract:
Omitted variables are one of the most important threats to the identification of causal effects. Several widely used methods assess the impact of omitted variables on empirical conclusions by comparing measures of selection on observables with measures of selection on unobservables. The recent literature has discussed various limitations of these existing methods, however. This includes a companio…
▽ More
Omitted variables are one of the most important threats to the identification of causal effects. Several widely used methods assess the impact of omitted variables on empirical conclusions by comparing measures of selection on observables with measures of selection on unobservables. The recent literature has discussed various limitations of these existing methods, however. This includes a companion paper of ours which explains issues that arise when the omitted variables are endogenous, meaning that they are correlated with the included controls. In the present paper, we develop a new approach to sensitivity analysis that avoids those limitations, while still allowing researchers to calibrate sensitivity parameters by comparing the magnitude of selection on observables with the magnitude of selection on unobservables as in previous methods. We illustrate our results in an empirical study of the effect of historical American frontier life on modern cultural beliefs. Finally, we implement these methods in the companion Stata module regsensitivity for easy use in practice.
△ Less
Submitted 20 June, 2025; v1 submitted 5 June, 2022;
originally announced June 2022.
-
Choosing Exogeneity Assumptions in Potential Outcome Models
Authors:
Matthew A. Masten,
Alexandre Poirier
Abstract:
There are many kinds of exogeneity assumptions. How should researchers choose among them? When exogeneity is imposed on an unobservable like a potential outcome, we argue that the form of exogeneity should be chosen based on the kind of selection on unobservables it allows. Consequently, researchers can assess the plausibility of any exogeneity assumption by studying the distributions of treatment…
▽ More
There are many kinds of exogeneity assumptions. How should researchers choose among them? When exogeneity is imposed on an unobservable like a potential outcome, we argue that the form of exogeneity should be chosen based on the kind of selection on unobservables it allows. Consequently, researchers can assess the plausibility of any exogeneity assumption by studying the distributions of treatment given the unobservables that are consistent with that assumption. We use this approach to study two common exogeneity assumptions: quantile and mean independence. We show that both assumptions require a kind of non-monotonic relationship between treatment and the potential outcomes. We discuss how to assess the plausibility of this kind of treatment selection. We also show how to define a new and weaker version of quantile independence that allows for monotonic treatment selection. We then show the implications of the choice of exogeneity assumption for identification. We apply these results in an empirical illustration of the effect of child soldiering on wages.
△ Less
Submitted 4 May, 2022;
originally announced May 2022.
-
Identification and Estimation of Partial Effects in Nonlinear Semiparametric Panel Models
Authors:
Laura Liu,
Alexandre Poirier,
Ji-Liang Shiu
Abstract:
Average partial effects (APEs) are often not point identified in panel models with unrestricted unobserved individual heterogeneity, such as a binary response panel model with fixed effects and logistic errors as a special case. This lack of point identification occurs despite the identification of these models' common coefficients. We provide a unified framework to establish the point identificat…
▽ More
Average partial effects (APEs) are often not point identified in panel models with unrestricted unobserved individual heterogeneity, such as a binary response panel model with fixed effects and logistic errors as a special case. This lack of point identification occurs despite the identification of these models' common coefficients. We provide a unified framework to establish the point identification of various partial effects in a wide class of nonlinear semiparametric models under an index sufficiency assumption on the unobserved heterogeneity, even when the error distribution is unspecified and non-stationary. This assumption does not impose parametric restrictions on the unobserved heterogeneity and idiosyncratic errors. We also present partial identification results when the support condition fails. We then propose three-step semiparametric estimators for APEs, average structural functions, and average marginal effects, and show their consistency and asymptotic normality. Finally, we illustrate our approach in a study of determinants of married women's labor supply.
△ Less
Submitted 30 July, 2024; v1 submitted 26 May, 2021;
originally announced May 2021.
-
Assessing Sensitivity to Unconfoundedness: Estimation and Inference
Authors:
Matthew A. Masten,
Alexandre Poirier,
Linqi Zhang
Abstract:
This paper provides a set of methods for quantifying the robustness of treatment effects estimated using the unconfoundedness assumption (also known as selection on observables or conditional independence). Specifically, we estimate and do inference on bounds on various treatment effect parameters, like the average treatment effect (ATE) and the average effect of treatment on the treated (ATT), un…
▽ More
This paper provides a set of methods for quantifying the robustness of treatment effects estimated using the unconfoundedness assumption (also known as selection on observables or conditional independence). Specifically, we estimate and do inference on bounds on various treatment effect parameters, like the average treatment effect (ATE) and the average effect of treatment on the treated (ATT), under nonparametric relaxations of the unconfoundedness assumption indexed by a scalar sensitivity parameter c. These relaxations allow for limited selection on unobservables, depending on the value of c. For large enough c, these bounds equal the no assumptions bounds. Using a non-standard bootstrap method, we show how to construct confidence bands for these bound functions which are uniform over all values of c. We illustrate these methods with an empirical application to effects of the National Supported Work Demonstration program. We implement these methods in a companion Stata module for easy use in practice.
△ Less
Submitted 31 December, 2020;
originally announced December 2020.
-
Salvaging Falsified Instrumental Variable Models
Authors:
Matthew A. Masten,
Alexandre Poirier
Abstract:
What should researchers do when their baseline model is refuted? We provide four constructive answers. First, researchers can measure the extent of falsification. To do this, we consider continuous relaxations of the baseline assumptions of concern. We then define the falsification frontier: The smallest relaxations of the baseline model which are not refuted. This frontier provides a quantitative…
▽ More
What should researchers do when their baseline model is refuted? We provide four constructive answers. First, researchers can measure the extent of falsification. To do this, we consider continuous relaxations of the baseline assumptions of concern. We then define the falsification frontier: The smallest relaxations of the baseline model which are not refuted. This frontier provides a quantitative measure of the extent of falsification. Second, researchers can present the identified set for the parameter of interest under the assumption that the true model lies somewhere on this frontier. We call this the falsification adaptive set. This set generalizes the standard baseline estimand to account for possible falsification. Third, researchers can present the identified set for a specific point on this frontier. Finally, as a sensitivity analysis, researchers can present identified sets for points beyond the frontier. To illustrate these four ways of salvaging falsified models, we study overidentifying restrictions in two instrumental variable models: a homogeneous effects linear model, and heterogeneous effect models with either binary or continuous outcomes. In the linear model, we consider the classical overidentifying restrictions implied when multiple instruments are observed. We generalize these conditions by considering continuous relaxations of the classical exclusion restrictions. By sufficiently weakening the assumptions, a falsified baseline model becomes non-falsified. We obtain analogous results in the heterogeneous effect models, where we derive identified sets for marginal distributions of potential outcomes, falsification frontiers, and falsification adaptive sets under continuous relaxations of the instrument exogeneity assumptions. We illustrate our results in four different empirical applications.
△ Less
Submitted 6 January, 2020; v1 submitted 30 December, 2018;
originally announced December 2018.
-
Interpreting Quantile Independence
Authors:
Matthew A. Masten,
Alexandre Poirier
Abstract:
How should one assess the credibility of assumptions weaker than statistical independence, like quantile independence? In the context of identifying causal effects of a treatment variable, we argue that such deviations should be chosen based on the form of selection on unobservables they allow. For quantile independence, we characterize this form of treatment selection. Specifically, we show that…
▽ More
How should one assess the credibility of assumptions weaker than statistical independence, like quantile independence? In the context of identifying causal effects of a treatment variable, we argue that such deviations should be chosen based on the form of selection on unobservables they allow. For quantile independence, we characterize this form of treatment selection. Specifically, we show that quantile independence is equivalent to a constraint on the average value of either a latent propensity score (for a binary treatment) or the cdf of treatment given the unobservables (for a continuous treatment). In both cases, this average value constraint requires a kind of non-monotonic treatment selection. Using these results, we show that several common treatment selection models are incompatible with quantile independence. We introduce a class of assumptions which weakens quantile independence by removing the average value constraint, and therefore allows for monotonic treatment selection. In a potential outcomes model with a binary treatment, we derive identified sets for the ATT and QTT under both classes of assumptions. In a numerical example we show that the average value constraint inherent in quantile independence has substantial identifying power. Our results suggest that researchers should carefully consider the credibility of this non-monotonicity property when using quantile independence to weaken full independence.
△ Less
Submitted 29 April, 2018;
originally announced April 2018.
-
Identification of Treatment Effects under Conditional Partial Independence
Authors:
Matthew A. Masten,
Alexandre Poirier
Abstract:
Conditional independence of treatment assignment from potential outcomes is a commonly used but nonrefutable assumption. We derive identified sets for various treatment effect parameters under nonparametric deviations from this conditional independence assumption. These deviations are defined via a conditional treatment assignment probability, which makes it straightforward to interpret. Our resul…
▽ More
Conditional independence of treatment assignment from potential outcomes is a commonly used but nonrefutable assumption. We derive identified sets for various treatment effect parameters under nonparametric deviations from this conditional independence assumption. These deviations are defined via a conditional treatment assignment probability, which makes it straightforward to interpret. Our results can be used to assess the robustness of empirical conclusions obtained under the baseline conditional independence assumption.
△ Less
Submitted 29 July, 2017;
originally announced July 2017.
-
Inference on Breakdown Frontiers
Authors:
Matthew A. Masten,
Alexandre Poirier
Abstract:
Given a set of baseline assumptions, a breakdown frontier is the boundary between the set of assumptions which lead to a specific conclusion and those which do not. In a potential outcomes model with a binary treatment, we consider two conclusions: First, that ATE is at least a specific value (e.g., nonnegative) and second that the proportion of units who benefit from treatment is at least a speci…
▽ More
Given a set of baseline assumptions, a breakdown frontier is the boundary between the set of assumptions which lead to a specific conclusion and those which do not. In a potential outcomes model with a binary treatment, we consider two conclusions: First, that ATE is at least a specific value (e.g., nonnegative) and second that the proportion of units who benefit from treatment is at least a specific value (e.g., at least 50\%). For these conclusions, we derive the breakdown frontier for two kinds of assumptions: one which indexes relaxations of the baseline random assignment of treatment assumption, and one which indexes relaxations of the baseline rank invariance assumption. These classes of assumptions nest both the point identifying assumptions of random assignment and rank invariance and the opposite end of no constraints on treatment selection or the dependence structure between potential outcomes. This frontier provides a quantitative measure of robustness of conclusions to relaxations of the baseline point identifying assumptions. We derive $\sqrt{N}$-consistent sample analog estimators for these frontiers. We then provide two asymptotically valid bootstrap procedures for constructing lower uniform confidence bands for the breakdown frontier. As a measure of robustness, estimated breakdown frontiers and their corresponding confidence bands can be presented alongside traditional point estimates and confidence intervals obtained under point identifying assumptions. We illustrate this approach in an empirical application to the effect of child soldiering on wages. We find that sufficiently weak conclusions are robust to simultaneous failures of rank invariance and random assignment, while some stronger conclusions are fairly robust to failures of rank invariance but not necessarily to relaxations of random assignment.
△ Less
Submitted 2 February, 2019; v1 submitted 12 May, 2017;
originally announced May 2017.
