-
ARMA-Design: Optimal Treatment Allocation Strategies for A/B Testing in Partially Observable Time Series Experiments
Authors:
Ke Sun,
Linglong Kong,
Hongtu Zhu,
Chengchun Shi
Abstract:
Online experiments %in which experimental units receive a sequence of treatments over time are frequently employed in many technological companies to evaluate the performance of a newly developed policy, product, or treatment relative to a baseline control. In many applications, the experimental units receive a sequence of treatments over time. To handle these time-dependent settings, existing A/B…
▽ More
Online experiments %in which experimental units receive a sequence of treatments over time are frequently employed in many technological companies to evaluate the performance of a newly developed policy, product, or treatment relative to a baseline control. In many applications, the experimental units receive a sequence of treatments over time. To handle these time-dependent settings, existing A/B testing solutions typically assume a fully observable experimental environment that satisfies the Markov condition. However, this assumption often does not hold in practice.
This paper studies the optimal design for A/B testing in partially observable online experiments. We introduce a controlled (vector) autoregressive moving average model to capture partial observability. We introduce a small signal asymptotic framework to simplify the calculation of asymptotic mean squared errors of average treatment effect estimators under various designs. We develop two algorithms to estimate the optimal design: one utilizing constrained optimization and the other employing reinforcement learning. We demonstrate the superior performance of our designs using two dispatch simulators that realistically mimic the behaviors of drivers and passengers to create virtual environments, along with two real datasets from a ride-sharing company. A Python implementation of our proposal is available at https://github.com/datake/ARMADesign.
△ Less
Submitted 11 January, 2025; v1 submitted 9 August, 2024;
originally announced August 2024.
-
Weak (Proxy) Factors Robust Hansen-Jagannathan Distance For Linear Asset Pricing Models
Authors:
Lingwei Kong
Abstract:
The Hansen-Jagannathan (HJ) distance statistic is one of the most dominant measures of model misspecification. However, the conventional HJ specification test procedure has poor finite sample performance, and we show that it can be size distorted even in large samples when (proxy) factors exhibit small correlations with asset returns. In other words, applied researchers are likely to falsely rejec…
▽ More
The Hansen-Jagannathan (HJ) distance statistic is one of the most dominant measures of model misspecification. However, the conventional HJ specification test procedure has poor finite sample performance, and we show that it can be size distorted even in large samples when (proxy) factors exhibit small correlations with asset returns. In other words, applied researchers are likely to falsely reject a model even when it is correctly specified. We provide two alternatives for the HJ statistic and two corresponding novel procedures for model specification tests, which are robust against the presence of weak (proxy) factors, and we also offer a novel robust risk premia estimator. Simulation exercises support our theory. Our empirical application documents the non-reliability of the traditional HJ test since it may produce counter-intuitive results when comparing nested models by rejecting a four-factor model but not the reduced three-factor model. At the same time, our proposed methods are practically more appealing and show support for a four-factor model for Fama French portfolios.
△ Less
Submitted 26 July, 2023;
originally announced July 2023.
-
Identification Robust Inference for the Risk Premium in Term Structure Models
Authors:
Frank Kleibergen,
Lingwei Kong
Abstract:
We propose identification robust statistics for testing hypotheses on the risk premia in dynamic affine term structure models. We do so using the moment equation specification proposed for these models in Adrian et al. (2013). We extend the subset (factor) Anderson-Rubin test from Guggenberger et al. (2012) to models with multiple dynamic factors and time-varying risk prices. Unlike projection-bas…
▽ More
We propose identification robust statistics for testing hypotheses on the risk premia in dynamic affine term structure models. We do so using the moment equation specification proposed for these models in Adrian et al. (2013). We extend the subset (factor) Anderson-Rubin test from Guggenberger et al. (2012) to models with multiple dynamic factors and time-varying risk prices. Unlike projection-based tests, it provides a computationally tractable manner to conduct identification robust tests on a larger number of parameters. We analyze the potential identification issues arising in empirical studies. Statistical inference based on the three-stage estimator from Adrian et al. (2013) requires knowledge of the factors' quality and is misleading without full-rank beta's or with sampling errors of comparable size as the loadings. Empirical applications show that some factors, though potentially weak, may drive the time variation of risk prices, and weak identification issues are more prominent in multi-factor models.
△ Less
Submitted 24 July, 2023;
originally announced July 2023.
