-
Mosaic inference on panel data
Authors:
Asher Spector,
Rina Foygel Barber,
Emmanuel Candès
Abstract:
Analysis of panel data via linear regression is widespread across disciplines. To perform statistical inference, such analyses typically assume that clusters of observations are jointly independent. For example, one might assume that observations in New York are independent of observations in New Jersey. Are such assumptions plausible? Might there be hidden dependencies between nearby clusters? Th…
▽ More
Analysis of panel data via linear regression is widespread across disciplines. To perform statistical inference, such analyses typically assume that clusters of observations are jointly independent. For example, one might assume that observations in New York are independent of observations in New Jersey. Are such assumptions plausible? Might there be hidden dependencies between nearby clusters? This paper introduces a mosaic permutation test that can (i) test the cluster-independence assumption and (ii) produce confidence intervals for linear models without assuming the full cluster-independence assumption. The key idea behind our method is to apply a permutation test to carefully constructed residual estimates that obey the same invariances as the true errors. As a result, our method yields finite-sample valid inferences under a mild "local exchangeability" condition. This condition differs from the typical cluster-independence assumption, as neither assumption implies the other. Furthermore, our method is asymptotically valid under cluster-independence (with no exchangeability assumptions). Together, these results show our method is valid under assumptions that are arguably weaker than the assumptions underlying many classical methods. In experiments on well-studied datasets from the literature, we find that many existing methods produce variance estimates that are up to five times too small, whereas mosaic methods produce reliable results. We implement our methods in the python package mosaicperm.
△ Less
Submitted 4 June, 2025;
originally announced June 2025.
-
Using Individualized Treatment Effects to Assess Treatment Effect Heterogeneity
Authors:
Konstantinos Sechidis,
Cong Zhang,
Sophie Sun,
Yao Chen,
Asher Spector,
Björn Bornkamp
Abstract:
Assessing treatment effect heterogeneity (TEH) in clinical trials is crucial, as it provides insights into the variability of treatment responses among patients, influencing important decisions related to drug development. Furthermore, it can lead to personalized medicine by tailoring treatments to individual patient characteristics. This paper introduces novel methodologies for assessing treatmen…
▽ More
Assessing treatment effect heterogeneity (TEH) in clinical trials is crucial, as it provides insights into the variability of treatment responses among patients, influencing important decisions related to drug development. Furthermore, it can lead to personalized medicine by tailoring treatments to individual patient characteristics. This paper introduces novel methodologies for assessing treatment effects using the individual treatment effect as a basis. To estimate this effect, we use a Double Robust (DR) learner to infer a pseudo-outcome that reflects the causal contrast. This pseudo-outcome is then used to perform three objectives: (1) a global test for heterogeneity, (2) ranking covariates based on their influence on effect modification, and (3) providing estimates of the individualized treatment effect. We compare our DR-learner with various alternatives and competing methods in a simulation study, and also use it to assess heterogeneity in a pooled analysis of five Phase III trials in psoriatic arthritis. By integrating these methods with the recently proposed WATCH workflow (Workflow to Assess Treatment Effect Heterogeneity in Drug Development for Clinical Trial Sponsors), we provide a robust framework for analyzing TEH, offering insights that enable more informed decision-making in this challenging area.
△ Less
Submitted 2 February, 2025;
originally announced February 2025.
-
The mosaic permutation test: an exact and nonparametric goodness-of-fit test for factor models
Authors:
Asher Spector,
Rina Foygel Barber,
Trevor Hastie,
Ronald N. Kahn,
Emmanuel Candès
Abstract:
Financial firms often rely on fundamental factor models to explain correlations among asset returns and manage risk. Yet after major events, e.g., COVID-19, analysts may reassess whether existing risk models continue to fit well: specifically, after accounting for a set of known factor exposures, are the residuals of the asset returns independent? With this motivation, we introduce the mosaic perm…
▽ More
Financial firms often rely on fundamental factor models to explain correlations among asset returns and manage risk. Yet after major events, e.g., COVID-19, analysts may reassess whether existing risk models continue to fit well: specifically, after accounting for a set of known factor exposures, are the residuals of the asset returns independent? With this motivation, we introduce the mosaic permutation test, a nonparametric goodness-of-fit test for preexisting factor models. Our method can leverage modern machine learning techniques to detect model violations while provably controlling the false positive rate, i.e., the probability of rejecting a well-fitting model, without making asymptotic approximations or parametric assumptions. This property helps prevent analysts from unnecessarily rebuilding accurate models, which can waste resources and increase risk. To illustrate our methodology, we apply the mosaic permutation test to the BlackRock Fundamental Equity Risk (BFRE) model. Although the BFRE model generally explains the most significant correlations among assets, we find evidence of unexplained correlations among certain real estate stocks, and we show that adding new factors improves model fit. We implement our methods in the python package mosaicperm.
△ Less
Submitted 26 September, 2024; v1 submitted 23 April, 2024;
originally announced April 2024.
-
Model-Agnostic Covariate-Assisted Inference on Partially Identified Causal Effects
Authors:
Wenlong Ji,
Lihua Lei,
Asher Spector
Abstract:
Many causal estimands are only partially identifiable since they depend on the unobservable joint distribution between potential outcomes. Stratification on pretreatment covariates can yield sharper bounds; however, unless the covariates are discrete with relatively small support, this approach typically requires binning covariates or estimating the conditional distributions of the potential outco…
▽ More
Many causal estimands are only partially identifiable since they depend on the unobservable joint distribution between potential outcomes. Stratification on pretreatment covariates can yield sharper bounds; however, unless the covariates are discrete with relatively small support, this approach typically requires binning covariates or estimating the conditional distributions of the potential outcomes given the covariates. Binning can result in substantial efficiency loss and become challenging to implement, even with a moderate number of covariates. Estimating conditional distributions, on the other hand, may yield invalid inference if the distributions are inaccurately estimated, such as when a misspecified model is used or when the covariates are high-dimensional. In this paper, we propose a unified and model-agnostic inferential approach for a wide class of partially identified estimands. Our method, based on duality theory for optimal transport problems, has four key properties. First, in randomized experiments, our approach can wrap around any estimates of the conditional distributions and provide uniformly valid inference, even if the initial estimates are arbitrarily inaccurate. A simple extension of our method to observational studies is doubly robust in the usual sense. Second, if nuisance parameters are estimated at semiparametric rates, our estimator is asymptotically unbiased for the sharp partial identification bound. Third, we can apply the multiplier bootstrap to select covariates and models without sacrificing validity, even if the true model is not selected. Finally, our method is computationally efficient. Overall, in three empirical applications, our method consistently reduces the width of estimated identified sets and confidence intervals without making additional structural assumptions.
△ Less
Submitted 17 November, 2024; v1 submitted 12 October, 2023;
originally announced October 2023.
-
Asymptotically Optimal Knockoff Statistics via the Masked Likelihood Ratio
Authors:
Asher Spector,
William Fithian
Abstract:
In feature selection problems, knockoffs are synthetic controls for the original features. Employing knockoffs allows analysts to use nearly any variable importance measure or "feature statistic" to select features while rigorously controlling false positives. However, it is not clear which statistic maximizes power. In this paper, we argue that state-of-the-art lasso-based feature statistics ofte…
▽ More
In feature selection problems, knockoffs are synthetic controls for the original features. Employing knockoffs allows analysts to use nearly any variable importance measure or "feature statistic" to select features while rigorously controlling false positives. However, it is not clear which statistic maximizes power. In this paper, we argue that state-of-the-art lasso-based feature statistics often prioritize features that are unlikely to be discovered, leading to low power in real applications. Instead, we introduce masked likelihood ratio (MLR) statistics, which prioritize features according to one's ability to distinguish each feature from its knockoff. Although no single feature statistic is uniformly most powerful in all situations, we show that MLR statistics asymptotically maximize the number of discoveries under a user-specified Bayesian model of the data. (Like all feature statistics, MLR statistics always provide frequentist error control.) This result places no restrictions on the problem dimensions and makes no parametric assumptions; instead, we require a "local dependence" condition that depends only on known quantities. In simulations and three real applications, MLR statistics outperform state-of-the-art feature statistics, including in settings where the Bayesian model is misspecified. We implement MLR statistics in the python package knockpy; our implementation is often faster than computing a cross-validated lasso.
△ Less
Submitted 1 October, 2024; v1 submitted 16 December, 2022;
originally announced December 2022.
-
Controlled Discovery and Localization of Signals via Bayesian Linear Programming
Authors:
Asher Spector,
Lucas Janson
Abstract:
Scientists often must simultaneously localize and discover signals. For instance, in genetic fine-mapping, high correlations between nearby genetic variants make it hard to identify the exact locations of causal variants. So the statistical task is to output as many disjoint regions containing a signal as possible, each as small as possible, while controlling false positives. Similar problems aris…
▽ More
Scientists often must simultaneously localize and discover signals. For instance, in genetic fine-mapping, high correlations between nearby genetic variants make it hard to identify the exact locations of causal variants. So the statistical task is to output as many disjoint regions containing a signal as possible, each as small as possible, while controlling false positives. Similar problems arise in any application where signals cannot be perfectly localized, such as locating stars in astronomical surveys and changepoint detection in sequential data. Common Bayesian approaches to these problems involve computing a posterior distribution over signal locations. However, existing procedures to translate these posteriors into actual credible regions for the signals fail to capture all the information in the posterior, leading to lower power and (sometimes) inflated false discoveries. With this motivation, we introduce Bayesian Linear Programming (BLiP). Given a posterior distribution over signals, BLiP outputs credible regions for signals which verifiably nearly maximize expected power while controlling false positives. BLiP overcomes an extremely high-dimensional and nonconvex problem to verifiably nearly maximize expected power while controlling false positives. BLiP is very computationally efficient compared to the cost of computing the posterior and can wrap around nearly any Bayesian model and algorithm. Applying BLiP to existing state-of-the-art analyses of UK Biobank data (for genetic fine-mapping) and the Sloan Digital Sky Survey (for astronomical point source detection) increased power by 30-120% in just a few minutes of additional computation. BLiP is implemented in pyblip (Python) and blipr (R).
△ Less
Submitted 28 January, 2023; v1 submitted 31 March, 2022;
originally announced March 2022.
-
Powerful Knockoffs via Minimizing Reconstructability
Authors:
Asher Spector,
Lucas Janson
Abstract:
Model-X knockoffs allows analysts to perform feature selection using almost any machine learning algorithm while still provably controlling the expected proportion of false discoveries. To apply model-X knockoffs, one must construct synthetic variables, called knockoffs, which effectively act as controls during feature selection. The gold standard for constructing knockoffs has been to minimize th…
▽ More
Model-X knockoffs allows analysts to perform feature selection using almost any machine learning algorithm while still provably controlling the expected proportion of false discoveries. To apply model-X knockoffs, one must construct synthetic variables, called knockoffs, which effectively act as controls during feature selection. The gold standard for constructing knockoffs has been to minimize the mean absolute correlation (MAC) between features and their knockoffs, but, surprisingly, we prove this procedure can be powerless in extremely easy settings, including Gaussian linear models with correlated exchangeable features. The key problem is that minimizing the MAC creates strong joint dependencies between the features and knockoffs, which allow machine learning algorithms to partially or fully reconstruct the effect of the features on the response using the knockoffs. To improve the power of knockoffs, we propose generating knockoffs which minimize the reconstructability (MRC) of the features, and we demonstrate our proposal for Gaussian features by showing it is computationally efficient, robust, and powerful. We also prove that certain MRC knockoffs minimize a natural definition of estimation error in Gaussian linear models. Furthermore, in an extensive set of simulations, we find many settings with correlated features in which MRC knockoffs dramatically outperform MAC-minimizing knockoffs and no settings in which MAC-minimizing knockoffs outperform MRC knockoffs by more than a very slight margin. We implement our methods and a host of others from the knockoffs literature in a new open source python package knockpy.
△ Less
Submitted 28 June, 2021; v1 submitted 30 November, 2020;
originally announced November 2020.
