-
Doubly protected estimation for survival outcomes utilizing external controls for randomized clinical trials
Authors:
Chenyin Gao,
Shu Yang,
Mingyang Shan,
Wenyu Wendy Ye,
Ilya Lipkovich,
Douglas Faries
Abstract:
Censored survival data are common in clinical trials, but small control groups can pose challenges, particularly in rare diseases or where balanced randomization is impractical. Recent approaches leverage external controls from historical studies or real-world data to strengthen treatment evaluation for survival outcomes. However, using external controls directly may introduce biases due to data h…
▽ More
Censored survival data are common in clinical trials, but small control groups can pose challenges, particularly in rare diseases or where balanced randomization is impractical. Recent approaches leverage external controls from historical studies or real-world data to strengthen treatment evaluation for survival outcomes. However, using external controls directly may introduce biases due to data heterogeneity. We propose a doubly protected estimator for the treatment-specific restricted mean survival time difference that is more efficient than trial-only estimators and mitigates biases from external data. Our method adjusts for covariate shifts via doubly robust estimation and addresses outcome drift using the DR-Learner for selective borrowing. The approach can incorporate machine learning to approximate survival curves and detect outcome drifts without strict parametric assumptions, borrowing only comparable external controls. Extensive simulation studies and a real-data application evaluating the efficacy of Galcanezumab in mitigating migraine headaches have been conducted to illustrate the effectiveness of our proposed framework.
△ Less
Submitted 14 May, 2025; v1 submitted 23 October, 2024;
originally announced October 2024.
-
Real Effect or Bias? Best Practices for Evaluating the Robustness of Real-World Evidence through Quantitative Sensitivity Analysis for Unmeasured Confounding
Authors:
Douglas Faries,
Chenyin Gao,
Xiang Zhang,
Chad Hazlett,
James Stamey,
Shu Yang,
Peng Ding,
Mingyang Shan,
Kristin Sheffield,
Nancy Dreyer
Abstract:
The assumption of no unmeasured confounders is a critical but unverifiable assumption required for causal inference yet quantitative sensitivity analyses to assess robustness of real-world evidence remains underutilized. The lack of use is likely in part due to complexity of implementation and often specific and restrictive data requirements required for application of each method. With the advent…
▽ More
The assumption of no unmeasured confounders is a critical but unverifiable assumption required for causal inference yet quantitative sensitivity analyses to assess robustness of real-world evidence remains underutilized. The lack of use is likely in part due to complexity of implementation and often specific and restrictive data requirements required for application of each method. With the advent of sensitivity analyses methods that are broadly applicable in that they do not require identification of a specific unmeasured confounder, along with publicly available code for implementation, roadblocks toward broader use are decreasing. To spur greater application, here we present a best practice guidance to address the potential for unmeasured confounding at both the design and analysis stages, including a set of framing questions and an analytic toolbox for researchers. The questions at the design stage guide the research through steps evaluating the potential robustness of the design while encouraging gathering of additional data to reduce uncertainty due to potential confounding. At the analysis stage, the questions guide researchers to quantifying the robustness of the observed result and providing researchers with a clearer indication of the robustness of their conclusions. We demonstrate the application of the guidance using simulated data based on a real-world fibromyalgia study, applying multiple methods from our analytic toolbox for illustration purposes.
△ Less
Submitted 13 September, 2023;
originally announced September 2023.
-
Sensitivity Analysis for Unmeasured Confounding in Medical Product Development and Evaluation Using Real World Evidence
Authors:
Peng Ding,
Yixin Fang,
Doug Faries,
Susan Gruber,
Hana Lee,
Joo-Yeon Lee,
Pallavi Mishra-Kalyani,
Mingyang Shan,
Mark van der Laan,
Shu Yang,
Xiang Zhang
Abstract:
The American Statistical Association Biopharmaceutical Section (ASA BIOP) working group on real-world evidence (RWE) has been making continuous, extended effort towards a goal of supporting and advancing regulatory science with respect to non-interventional, clinical studies intended to use real-world data for evidence generation for the purpose of medical product development and evaluation (i.e.,…
▽ More
The American Statistical Association Biopharmaceutical Section (ASA BIOP) working group on real-world evidence (RWE) has been making continuous, extended effort towards a goal of supporting and advancing regulatory science with respect to non-interventional, clinical studies intended to use real-world data for evidence generation for the purpose of medical product development and evaluation (i.e., RWE studies). In 2023, the working group published a manuscript delineating challenges and opportunities in constructing estimands for RWE studies following a framework in ICH E9(R1) guidance on estimand and sensitivity analysis. As a follow-up task, we describe the other issue in RWE studies, sensitivity analysis. Focusing on the issue of unmeasured confounding, we review availability and applicability of sensitivity analysis methods for different types unmeasured confounding. We discuss consideration on the choice and use of sensitivity analysis for RWE studies. Updated version of this article will present how findings from sensitivity analysis could support regulatory decision-making using a real example.
△ Less
Submitted 14 July, 2023;
originally announced July 2023.
-
Improving randomized controlled trial analysis via data-adaptive borrowing
Authors:
Chenyin Gao,
Shu Yang,
Mingyang Shan,
Wenyu Ye,
Ilya Lipkovich,
Douglas Faries
Abstract:
In recent years, real-world external controls have grown in popularity as a tool to empower randomized placebo-controlled trials, particularly in rare diseases or cases where balanced randomization is unethical or impractical. However, as external controls are not always comparable to the trials, direct borrowing without scrutiny may heavily bias the treatment effect estimator. Our paper proposes…
▽ More
In recent years, real-world external controls have grown in popularity as a tool to empower randomized placebo-controlled trials, particularly in rare diseases or cases where balanced randomization is unethical or impractical. However, as external controls are not always comparable to the trials, direct borrowing without scrutiny may heavily bias the treatment effect estimator. Our paper proposes a data-adaptive integrative framework capable of preventing unknown biases of the external controls. The adaptive nature is achieved by dynamically sorting out a comparable subset of the external controls via bias penalization. Our proposed method can simultaneously achieve (a) the semiparametric efficiency bound when the external controls are comparable and (b) selective borrowing that mitigates the impact of the existence of incomparable external controls. Furthermore, we establish statistical guarantees, including consistency, asymptotic distribution, and inference, providing type-I error control and good power. Extensive simulations and two real-data applications show that the proposed method leads to improved performance over the trial-only estimator across various bias-generating scenarios.
△ Less
Submitted 12 November, 2024; v1 submitted 28 June, 2023;
originally announced June 2023.
-
Double Machine Learning Methods for Estimating Average Treatment Effects: A Comparative Study
Authors:
Xiaoqing Tan,
Shu Yang,
Wenyu Ye,
Douglas E. Faries,
Ilya Lipkovich,
Zbigniew Kadziola
Abstract:
Observational cohort studies are increasingly being used for comparative effectiveness research to assess the safety of therapeutics. Recently, various doubly robust methods have been proposed for average treatment effect estimation by combining the treatment model and the outcome model via different vehicles, such as matching, weighting, and regression. The key advantage of doubly robust estimato…
▽ More
Observational cohort studies are increasingly being used for comparative effectiveness research to assess the safety of therapeutics. Recently, various doubly robust methods have been proposed for average treatment effect estimation by combining the treatment model and the outcome model via different vehicles, such as matching, weighting, and regression. The key advantage of doubly robust estimators is that they require either the treatment model or the outcome model to be correctly specified to obtain a consistent estimator of average treatment effects, and therefore lead to a more accurate and often more precise inference. However, little work has been done to understand how doubly robust estimators differ due to their unique strategies of using the treatment and outcome models and how machine learning techniques can be combined to boost their performance, which we call double machine learning estimators. Here we examine multiple popular doubly robust methods and compare their performance using different treatment and outcome modeling via extensive simulations and a real-world application. We found that incorporating machine learning with doubly robust estimators such as the targeted maximum likelihood estimator gives the best overall performance. Practical guidance on how to apply doubly robust estimators is provided.
△ Less
Submitted 9 March, 2025; v1 submitted 22 April, 2022;
originally announced April 2022.
-
Propensity score matching for estimating a marginal hazard ratio
Authors:
Tongrong Wang,
Honghe Zhao,
Shu Yang,
Shuhan Tang,
Zhanglin Cui,
Li Li,
Douglas E. Faries
Abstract:
Propensity score matching is commonly used to draw causal inference from observational survival data. However, its asymptotic properties have yet to be established, and variance estimation is still open to debate. We derive the statistical properties of the propensity score matching estimator of the marginal causal hazard ratio based on matching with replacement and a fixed number of matches. We a…
▽ More
Propensity score matching is commonly used to draw causal inference from observational survival data. However, its asymptotic properties have yet to be established, and variance estimation is still open to debate. We derive the statistical properties of the propensity score matching estimator of the marginal causal hazard ratio based on matching with replacement and a fixed number of matches. We also propose a double-resampling technique for variance estimation that takes into account the uncertainty due to propensity score estimation prior to matching.
△ Less
Submitted 31 January, 2024; v1 submitted 27 November, 2019;
originally announced November 2019.
-
Propensity Score Matching and Subclassification in Observational Studies with Multi-level Treatments
Authors:
Shu Yang,
Guido W. Imbens,
Zhanglin Cui,
Douglas Faries,
Zbigniew Kadziola
Abstract:
In this paper, we develop new methods for estimating average treatment effects in observational studies, focusing on settings with more than two treatment levels under unconfoundedness given pre-treatment variables. We emphasize subclassification and matching methods which have been found to be effective in the binary treatment literature and which are among the most popular methods in that settin…
▽ More
In this paper, we develop new methods for estimating average treatment effects in observational studies, focusing on settings with more than two treatment levels under unconfoundedness given pre-treatment variables. We emphasize subclassification and matching methods which have been found to be effective in the binary treatment literature and which are among the most popular methods in that setting. Whereas the literature has suggested that these particular propensity-based methods do not naturally extend to the multi-level treatment case, we show, using the concept of weak unconfoundedness, that adjusting for or matching on a scalar function of the pre-treatment variables removes all biases associated with observed pre-treatment variables. We apply the proposed methods to an analysis of the effect of treatments for fibromyalgia. We also carry out a simulation study to assess the finite sample performance of the methods relative to previously proposed methods.
△ Less
Submitted 14 December, 2015; v1 submitted 27 August, 2015;
originally announced August 2015.
