Showing 1–42 of 42 results for author: Zhu, K

Validating remotely sensed biomass estimates with forest inventory data in the western US

Authors: Xiuyu Cao, Joseph O. Sexton, Panshi Wang, Dimitrios Gounaridis, Neil H. Carter, Kai Zhu

Abstract: Monitoring aboveground biomass (AGB) and its density (AGBD) at high resolution is essential for carbon accounting and ecosystem management. While NASA's spaceborne Global Ecosystem Dynamics Investigation (GEDI) LiDAR mission provides globally distributed reference measurements for AGBD estimation, the majority of commercial remote sensing products based on GEDI remain without rigorous or independe… ▽ More Monitoring aboveground biomass (AGB) and its density (AGBD) at high resolution is essential for carbon accounting and ecosystem management. While NASA's spaceborne Global Ecosystem Dynamics Investigation (GEDI) LiDAR mission provides globally distributed reference measurements for AGBD estimation, the majority of commercial remote sensing products based on GEDI remain without rigorous or independent validation. Here, we present an independent regional validation of an AGBD dataset offered by terraPulse, Inc., based on independent reference data from the US Forest Service Forest Inventory and Analysis (FIA) program. Aggregated to 64,000-hectare hexagons and US counties across the US states of Utah, Nevada, and Washington, we found very strong agreement between terraPulse and FIA estimates. At the hexagon scale, we report R2 = 0.88, RMSE = 26.68 Mg/ha, and a correlation coefficient (r) of 0.94. At the county scale, agreement improves to R2 = 0.90, RMSE =32.62 Mg/ha, slope = 1.07, and r = 0.95. Spatial and statistical analyses indicated that terraPulse AGBD values tended to exceed FIA estimates in non-forest areas, likely due to FIA's limited sampling of non-forest vegetation. The terraPulse AGBD estimates also exhibited lower values in high-biomass forests, likely due to saturation effects in its optical remote-sensing covariates. This study advances operational carbon monitoring by delivering a scalable framework for comprehensive AGBD validation using independent FIA data, as well as a benchmark validation of a new commercial dataset for global biomass monitoring. △ Less

Submitted 3 June, 2025; originally announced June 2025.

Comments: 32 pages, 5 figures

Doubly Robust Fusion of Many Treatments for Policy Learning

Authors: Ke Zhu, Jianing Chu, Ilya Lipkovich, Wenyu Ye, Shu Yang

Abstract: Individualized treatment rules/recommendations (ITRs) aim to improve patient outcomes by tailoring treatments to the characteristics of each individual. However, when there are many treatment groups, existing methods face significant challenges due to data sparsity within treatment groups and highly unbalanced covariate distributions across groups. To address these challenges, we propose a novel c… ▽ More Individualized treatment rules/recommendations (ITRs) aim to improve patient outcomes by tailoring treatments to the characteristics of each individual. However, when there are many treatment groups, existing methods face significant challenges due to data sparsity within treatment groups and highly unbalanced covariate distributions across groups. To address these challenges, we propose a novel calibration-weighted treatment fusion procedure that robustly balances covariates across treatment groups and fuses similar treatments using a penalized working model. The fusion procedure ensures the recovery of latent treatment group structures when either the calibration model or the outcome model is correctly specified. In the fused treatment space, practitioners can seamlessly apply state-of-the-art ITR learning methods with the flexibility to utilize a subset of covariates, thereby achieving robustness while addressing practical concerns such as fairness. We establish theoretical guarantees, including consistency, the oracle property of treatment fusion, and regret bounds when integrated with multi-armed ITR learning methods such as policy trees. Simulation studies show superior group recovery and policy value compared to existing approaches. We illustrate the practical utility of our method using a nationwide electronic health record-derived de-identified database containing data from patients with Chronic Lymphocytic Leukemia and Small Lymphocytic Lymphoma. △ Less

Submitted 23 May, 2025; v1 submitted 12 May, 2025; originally announced May 2025.

Comments: Accepted by ICML 2025

Robust Estimation and Inference in Hybrid Controlled Trials for Binary Outcomes: A Case Study on Non-Small Cell Lung Cancer

Authors: Jiajun Liu, Ke Zhu, Shu Yang, Xiaofei Wang

Abstract: Hybrid controlled trials (HCTs), which augment randomized controlled trials (RCTs) with external controls (ECs), are increasingly receiving attention as a way to address limited power, slow accrual, and ethical concerns in clinical research. However, borrowing from ECs raises critical statistical challenges in estimation and inference, especially for binary outcomes where hidden bias is harder to… ▽ More Hybrid controlled trials (HCTs), which augment randomized controlled trials (RCTs) with external controls (ECs), are increasingly receiving attention as a way to address limited power, slow accrual, and ethical concerns in clinical research. However, borrowing from ECs raises critical statistical challenges in estimation and inference, especially for binary outcomes where hidden bias is harder to detect and estimands such as risk difference, risk ratio, and odds ratio are of primary interest. We propose a novel framework that combines doubly robust estimators for various estimands under covariate shift of ECs with conformal selective borrowing (CSB) to address outcome incomparability. CSB uses conformal inference with nearest-neighbor-based conformal scores and their label-conditional extensions to perform finite-sample exact individual-level EC selection, addressing the limited information in binary outcomes. To ensure strict type I error rate control for testing treatment effects while gaining power, we use a Fisher randomization test with the CSB estimator as the test statistic. Extensive simulations demonstrate the robust performance of our methods. We apply our method to data from CALGB 9633 and the National Cancer Database to evaluate chemotherapy effects in Stage IB non-small-cell lung cancer patients and show that the proposed method effectively mitigates hidden bias introduced by full-borrowing approaches, strictly controls the type I error rate, and improves the power over RCT-only analysis. △ Less

Submitted 30 April, 2025; originally announced May 2025.

Rejoinder to Reader Reaction "On exact randomization-based covariate-adjusted confidence intervals" by Jacob Fiksel

Authors: Ke Zhu, Hanzhong Liu

Abstract: We applaud Fiksel (2024) for their valuable contributions to randomization-based inference, particularly their work on inverting the Fisher randomization test (FRT) to construct confidence intervals using the covariate-adjusted test statistic. FRT is advocated by many scholars because it produces finite-sample exact p-values for any test statistic and can be easily adopted for any experimental des… ▽ More We applaud Fiksel (2024) for their valuable contributions to randomization-based inference, particularly their work on inverting the Fisher randomization test (FRT) to construct confidence intervals using the covariate-adjusted test statistic. FRT is advocated by many scholars because it produces finite-sample exact p-values for any test statistic and can be easily adopted for any experimental design (Rosenberger et al., 2019; Proschan and Dodd, 2019; Young, 2019; Bind and Rubin, 2020). By inverting FRTs, we can construct the randomization-based confidence interval (RBCI). To the best of our knowledge, Zhu and Liu (2023) are the first to analytically invert the FRT for the difference-in-means statistic. Fiksel (2024) extended this analytical approach to the covariate-adjusted statistic, producing a monotonic p-value function under certain conditions. In this rejoinder, we propose an analytical approach to invert the FRT for test statistics that yield a non-monotonic p-value function, with the studentized t-statistic as an important special case. Exploiting our analytical approach, we can recover the non-monotonic p-value function and construct RBCI based on the studentized t-statistic. The RBCI generated by the proposed analytical approach is guaranteed to achieve the desired coverage probability and resolve the contradiction between Luo et al. (2021) and Wu and Ding (2021). Simulation results validate our findings and demonstrate that our method is also computationally efficient. △ Less

Submitted 25 February, 2025; originally announced February 2025.

Comments: Published in Biometrics

Tensor dynamic conditional correlation model: A new way to pursuit "Holy Grail of investing"

Authors: Cheng Yu, Zhoufan Zhu, Ke Zhu

Abstract: Style investing creates asset classes (or the so-called "styles") with low correlations, aligning well with the principle of "Holy Grail of investing" in terms of portfolio selection. The returns of styles naturally form a tensor-valued time series, which requires new tools for studying the dynamics of the conditional correlation matrix to facilitate the aforementioned principle. Towards this goal… ▽ More Style investing creates asset classes (or the so-called "styles") with low correlations, aligning well with the principle of "Holy Grail of investing" in terms of portfolio selection. The returns of styles naturally form a tensor-valued time series, which requires new tools for studying the dynamics of the conditional correlation matrix to facilitate the aforementioned principle. Towards this goal, we introduce a new tensor dynamic conditional correlation (TDCC) model, which is based on two novel treatments: trace-normalization and dimension-normalization. These two normalizations adapt to the tensor nature of the data, and they are necessary except when the tensor data reduce to vector data. Moreover, we provide an easy-to-implement estimation procedure for the TDCC model, and examine its finite sample performance by simulations. Finally, we assess the usefulness of the TDCC model in international portfolio selection across ten global markets and in large portfolio selection for 1800 stocks from the Chinese stock market. △ Less

Submitted 19 February, 2025; originally announced February 2025.

Targeted Data Fusion for Causal Survival Analysis Under Distribution Shift

Authors: Yi Liu, Alexander W. Levis, Ke Zhu, Shu Yang, Peter B. Gilbert, Larry Han

Abstract: Causal inference across multiple data sources offers a promising avenue to enhance the generalizability and replicability of scientific findings. However, data integration methods for time-to-event outcomes, common in biomedical research, are underdeveloped. Existing approaches focus on binary or continuous outcomes but fail to address the unique challenges of survival analysis, such as censoring… ▽ More Causal inference across multiple data sources offers a promising avenue to enhance the generalizability and replicability of scientific findings. However, data integration methods for time-to-event outcomes, common in biomedical research, are underdeveloped. Existing approaches focus on binary or continuous outcomes but fail to address the unique challenges of survival analysis, such as censoring and the integration of discrete and continuous time. To bridge this gap, we propose two novel methods for estimating target site-specific causal effects in multi-source settings. First, we develop a semiparametric efficient estimator for settings where individual-level data can be shared across sites. Second, we introduce a federated learning framework designed for privacy-constrained environments, which dynamically reweights source-specific contributions to account for discrepancies with the target population. Both methods leverage flexible, nonparametric machine learning models to improve robustness and efficiency. We illustrate the utility of our approaches through simulation studies and an application to multi-site randomized trials of monoclonal neutralizing antibodies for HIV-1 prevention, conducted among cisgender men and transgender persons in the United States, Brazil, Peru, and Switzerland, as well as among women in sub-Saharan Africa. Our findings underscore the potential of these methods to enable efficient, privacy-preserving causal inference for time-to-event outcomes under distribution shift. △ Less

Submitted 14 May, 2025; v1 submitted 30 January, 2025; originally announced January 2025.

COADVISE: Covariate Adjustment with Variable Selection in Randomized Controlled Trials

Authors: Yi Liu, Ke Zhu, Larry Han, Shu Yang

Abstract: Adjusting for covariates in randomized controlled trials can enhance the credibility and efficiency of treatment effect estimation. However, handling numerous covariates and their complex (non-linear) transformations poses a challenge. Motivated by the case study of the Best Apnea Interventions for Research (BestAIR) trial data from the National Sleep Research Resource (NSRR), where the number of… ▽ More Adjusting for covariates in randomized controlled trials can enhance the credibility and efficiency of treatment effect estimation. However, handling numerous covariates and their complex (non-linear) transformations poses a challenge. Motivated by the case study of the Best Apnea Interventions for Research (BestAIR) trial data from the National Sleep Research Resource (NSRR), where the number of covariates (p=114) is comparable to the sample size (N=196), we propose a principled Covariate Adjustment with Variable Selection (COADVISE) framework. COADVISE enables variable selection for covariates most relevant to the outcome while accommodating both linear and nonlinear adjustments. This framework ensures consistent estimates with improved efficiency over unadjusted estimators and provides robust variance estimation, even under outcome model misspecification. We demonstrate efficiency gains through theoretical analysis, extensive simulations, and a re-analysis of the BestAIR trial data to compare alternative variable selection strategies, offering cautionary recommendations. A user-friendly R package, Coadvise, is available to facilitate practical implementation. △ Less

Submitted 26 February, 2025; v1 submitted 15 January, 2025; originally announced January 2025.

Imputation-based randomization tests for randomized experiments with interference

Authors: Tingxuan Han, Ke Zhu, Hanzhong Liu, Ke Deng

Abstract: The presence of interference renders classic Fisher randomization tests infeasible due to nuisance unknowns. To address this issue, we propose imputing the nuisance unknowns and computing Fisher randomization p-values multiple times, then averaging them. We term this approach the imputation-based randomization test and provide theoretical results on its asymptotic validity. Our method leverages th… ▽ More The presence of interference renders classic Fisher randomization tests infeasible due to nuisance unknowns. To address this issue, we propose imputing the nuisance unknowns and computing Fisher randomization p-values multiple times, then averaging them. We term this approach the imputation-based randomization test and provide theoretical results on its asymptotic validity. Our method leverages the merits of randomization and the flexibility of the Bayesian framework: for multiple imputations, we can either employ the empirical distribution of observed outcomes to achieve robustness against model mis-specification or utilize a parametric model to incorporate prior information. Simulation results demonstrate that our method effectively controls the type I error rate and significantly enhances the testing power compared to existing randomization tests for randomized experiments with interference. We apply our method to a two-round randomized experiment with multiple treatments and one-way interference, where existing randomization tests exhibit limited power. △ Less

Submitted 13 November, 2024; originally announced November 2024.

Comments: 39 pages, 12 figures

Enhancing Statistical Validity and Power in Hybrid Controlled Trials: A Randomization Inference Approach with Conformal Selective Borrowing

Authors: Ke Zhu, Shu Yang, Xiaofei Wang

Abstract: External controls from historical trials or observational data can augment randomized controlled trials when large-scale randomization is impractical or unethical, such as in drug evaluation for rare diseases. However, non-randomized external controls can introduce biases, and existing Bayesian and frequentist methods may inflate the type I error rate, particularly in small-sample trials where ext… ▽ More External controls from historical trials or observational data can augment randomized controlled trials when large-scale randomization is impractical or unethical, such as in drug evaluation for rare diseases. However, non-randomized external controls can introduce biases, and existing Bayesian and frequentist methods may inflate the type I error rate, particularly in small-sample trials where external data borrowing is most critical. To address these challenges, we propose a randomization inference framework that ensures finite-sample exact and model-free type I error rate control, adhering to the "analyze as you randomize" principle to safeguard against hidden biases. Recognizing that biased external controls reduce the power of randomization tests, we leverage conformal inference to develop an individualized test-then-pool procedure that selectively borrows comparable external controls to improve power. Our approach incorporates selection uncertainty into randomization tests, providing valid post-selection inference. Additionally, we propose an adaptive procedure to optimize the selection threshold by minimizing the mean squared error across a class of estimators encompassing both no-borrowing and full-borrowing approaches. The proposed methods are supported by non-asymptotic theoretical analysis, validated through simulations, and applied to a randomized lung cancer trial that integrates external controls from the National Cancer Database. △ Less

Submitted 7 May, 2025; v1 submitted 15 October, 2024; originally announced October 2024.

Comments: Accepted by ICML 2025

Enhancement of price trend trading strategies via image-induced importance weights

Authors: Zhoufan Zhu, Ke Zhu

Abstract: We open up the "black-box" to identify the predictive general price patterns in price chart images via the deep learning image analysis techniques. Our identified price patterns lead to the construction of image-induced importance (triple-I) weights, which are applied to weighted moving average the existing price trend trading signals according to their level of importance in predicting price move… ▽ More We open up the "black-box" to identify the predictive general price patterns in price chart images via the deep learning image analysis techniques. Our identified price patterns lead to the construction of image-induced importance (triple-I) weights, which are applied to weighted moving average the existing price trend trading signals according to their level of importance in predicting price movements. From an extensive empirical analysis on the Chinese stock market, we show that the triple-I weighting scheme can significantly enhance the price trend trading signals for proposing portfolios, with a thoughtful robustness study in terms of network specifications, image structures, and stock sizes. Moreover, we demonstrate that the triple-I weighting scheme is able to propose long-term portfolios from a time-scale transfer learning, enhance the news-based trading strategies through a non-technical transfer learning, and increase the overall strength of numerous trading rules for portfolio selection. △ Less

Submitted 15 August, 2024; originally announced August 2024.

Weighted Brier Score -- an Overall Summary Measure for Risk Prediction Models with Clinical Utility Consideration

Authors: Kehao Zhu, Yingye Zheng, Kwun Chuen Gary Chan

Abstract: As advancements in novel biomarker-based algorithms and models accelerate disease risk prediction and stratification in medicine, it is crucial to evaluate these models within the context of their intended clinical application. Prediction models output the absolute risk of disease; subsequently, patient counseling and shared decision-making are based on the estimated individual risk and cost-benef… ▽ More As advancements in novel biomarker-based algorithms and models accelerate disease risk prediction and stratification in medicine, it is crucial to evaluate these models within the context of their intended clinical application. Prediction models output the absolute risk of disease; subsequently, patient counseling and shared decision-making are based on the estimated individual risk and cost-benefit assessment. The overall impact of the application is often referred to as clinical utility, which received significant attention in terms of model assessment lately. The classic Brier score is a popular measure of prediction accuracy; however, it is insufficient for effectively assessing clinical utility. To address this limitation, we propose a class of weighted Brier scores that aligns with the decision-theoretic framework of clinical utility. Additionally, we decompose the weighted Brier score into discrimination and calibration components, examining how weighting influences the overall score and its individual components. Through this decomposition, we link the weighted Brier score to the $H$ measure, which has been proposed as a coherent alternative to the area under the receiver operating characteristic curve. This theoretical link to the $H$ measure further supports our weighting method and underscores the essential elements of discrimination and calibration in risk prediction evaluation. The practical use of the weighted Brier score as an overall summary is demonstrated using data from the Prostate Cancer Active Surveillance Study (PASS). △ Less

Submitted 2 August, 2024; originally announced August 2024.

Inference for the panel ARMA-GARCH model when both $N$ and $T$ are large

Authors: Bing Su, Ke Zhu

Abstract: We propose a panel ARMA-GARCH model to capture the dynamics of large panel data with $N$ individuals over $T$ time periods. For this model, we provide a two-step estimation procedure to estimate the ARMA parameters and GARCH parameters stepwisely. Under some regular conditions, we show that all of the proposed estimators are asymptotically normal with the convergence rate $(NT)^{-1/2}$, and they h… ▽ More We propose a panel ARMA-GARCH model to capture the dynamics of large panel data with $N$ individuals over $T$ time periods. For this model, we provide a two-step estimation procedure to estimate the ARMA parameters and GARCH parameters stepwisely. Under some regular conditions, we show that all of the proposed estimators are asymptotically normal with the convergence rate $(NT)^{-1/2}$, and they have the asymptotic biases when both $N$ and $T$ diverge to infinity at the same rate. Particularly, we find that the asymptotic biases result from the fixed effect, estimation effect, and unobservable initial values. To correct the biases, we further propose the bias-corrected version of estimators by using either the analytical asymptotics or jackknife method. Our asymptotic results are based on a new central limit theorem for the linear-quadratic form in the martingale difference sequence, when the weight matrix is uniformly bounded in row and column. Simulations and one real example are given to demonstrate the usefulness of our panel ARMA-GARCH model. △ Less

Submitted 28 April, 2024; originally announced April 2024.

Estimation for conditional moment models based on martingale difference divergence

Authors: Kunyang Song, Feiyu Jiang, Ke Zhu

Abstract: We provide a new estimation method for conditional moment models via the martingale difference divergence (MDD).Our MDD-based estimation method is formed in the framework of a continuum of unconditional moment restrictions. Unlike the existing estimation methods in this framework, the MDD-based estimation method adopts a non-integrable weighting function, which could grab more information from unc… ▽ More We provide a new estimation method for conditional moment models via the martingale difference divergence (MDD).Our MDD-based estimation method is formed in the framework of a continuum of unconditional moment restrictions. Unlike the existing estimation methods in this framework, the MDD-based estimation method adopts a non-integrable weighting function, which could grab more information from unconditional moment restrictions than the integrable weighting function to enhance the estimation efficiency. Due to the nature of shift-invariance in MDD, our MDD-based estimation method can not identify the intercept parameters. To overcome this identification issue, we further provide a two-step estimation procedure for the model with intercept parameters. Under regularity conditions, we establish the asymptotics of the proposed estimators, which are not only easy-to-implement with analytic asymptotic variances, but also applicable to time series data with an unspecified form of conditional heteroskedasticity. Finally, we illustrate the usefulness of the proposed estimators by simulations and two real examples. △ Less

Submitted 17 April, 2024; originally announced April 2024.

Sharp variance estimator and causal bootstrap in stratified randomized experiments

Authors: Haoyang Yu, Ke Zhu, Hanzhong Liu

Abstract: Randomized experiments are the gold standard for estimating treatment effects, and randomization serves as a reasoned basis for inference. In widely used stratified randomized experiments, randomization-based finite-population asymptotic theory enables valid inference for the average treatment effect, relying on normal approximation and a Neyman-type conservative variance estimator. However, when… ▽ More Randomized experiments are the gold standard for estimating treatment effects, and randomization serves as a reasoned basis for inference. In widely used stratified randomized experiments, randomization-based finite-population asymptotic theory enables valid inference for the average treatment effect, relying on normal approximation and a Neyman-type conservative variance estimator. However, when the sample size is small or the outcomes are skewed, the Neyman-type variance estimator may become overly conservative, and the normal approximation can fail. To address these issues, we propose a sharp variance estimator and two causal bootstrap methods to more accurately approximate the sampling distribution of the weighted difference-in-means estimator in stratified randomized experiments. The first causal bootstrap procedure is based on rank-preserving imputation and we prove its second-order refinement over normal approximation. The second causal bootstrap procedure is based on constant-treatment-effect imputation and is further applicable in paired experiments. In contrast to traditional bootstrap methods, where randomness originates from hypothetical super-population sampling, our analysis for the proposed causal bootstrap is randomization-based, relying solely on the randomness of treatment assignment in randomized experiments. Numerical studies and two real data applications demonstrate advantages of our proposed methods in finite samples. The \texttt{R} package \texttt{CausalBootstrap} implementing our method is publicly available. △ Less

Submitted 16 May, 2025; v1 submitted 29 January, 2024; originally announced January 2024.

Comments: Accepted by Statistics in Medicine

Global Inequality in Cooling from Urban Green Spaces and its Climate Change Adaptation Potential

Authors: Yuxiang Li, Jens-Christian Svenning, Weiqi Zhou, Kai Zhu, Jesse F. Abrams, Timothy M. Lenton, Shuqing N. Teng, Robert R. Dunn, Chi Xu

Abstract: Heat extremes are projected to severely impact humanity and with increasing geographic disparities. Global South countries are more exposed to heat extremes and have reduced adaptation capacity. One documented source of such adaptation inequality is a lack of resources to cool down indoor temperatures. Less is known about the capacity to ameliorate outdoor heat stress. Here, we assess global inequ… ▽ More Heat extremes are projected to severely impact humanity and with increasing geographic disparities. Global South countries are more exposed to heat extremes and have reduced adaptation capacity. One documented source of such adaptation inequality is a lack of resources to cool down indoor temperatures. Less is known about the capacity to ameliorate outdoor heat stress. Here, we assess global inequality in green infrastructure, on which urban residents critically rely to ameliorate lethal heat stress outdoors. We use satellite-derived indicators of land surface temperature and urban green space area to quantify the daytime cooling capacity of urban green spaces in the hottest months across ~500 cities with population size over 1 million per city globally. Our results show a striking contrast with an about two-fold lower cooling capacity in Global South cities compared to the Global North (2.1 degrees Celsius vs. 3.8 degrees Celsius). A similar gap occurs for the cooling adaptation benefits received by an average urban resident (Global South 1.9 degrees Celsius vs. North 3.6 degrees Celsius), i.e., accounting for relative spatial distributions of people and urban green spaces. This cooling adaptation inequality is attributed to the discrepancies in urban green space quantity and quality between Global North and South cities, jointly shaped by natural and socioeconomic factors. Our analyses suggest vast potential for enhancing outdoor cooling adaptation while reducing its global inequality through expanding and optimizing urban green infrastructure. △ Less

Submitted 18 July, 2023; originally announced July 2023.

Comments: 56 pages, 28 figures

Matrix GARCH Model: Inference and Application

Authors: Cheng Yu, Dong Li, Feiyu Jiang, Ke Zhu

Abstract: Matrix-variate time series data are largely available in applications. However, no attempt has been made to study their conditional heteroskedasticity that is often observed in economic and financial data. To address this gap, we propose a novel matrix generalized autoregressive conditional heteroskedasticity (GARCH) model to capture the dynamics of conditional row and column covariance matrices o… ▽ More Matrix-variate time series data are largely available in applications. However, no attempt has been made to study their conditional heteroskedasticity that is often observed in economic and financial data. To address this gap, we propose a novel matrix generalized autoregressive conditional heteroskedasticity (GARCH) model to capture the dynamics of conditional row and column covariance matrices of matrix time series. The key innovation of the matrix GARCH model is the use of a univariate GARCH specification for the trace of conditional row or column covariance matrix, which allows for the identification of conditional row and column covariance matrices. Moreover, we introduce a quasi maximum likelihood estimator (QMLE) for model estimation and develop a portmanteau test for model diagnostic checking. Simulation studies are conducted to assess the finite-sample performance of the QMLE and portmanteau test. To handle large dimensional matrix time series, we also propose a matrix factor GARCH model. Finally, we demonstrate the superiority of the matrix GARCH and matrix factor GARCH models over existing multivariate GARCH-type models in volatility forecasting and portfolio allocations using three applications on credit default swap prices, global stock sector indices, and future prices. △ Less

Submitted 8 June, 2023; originally announced June 2023.

Quantiled conditional variance, skewness, and kurtosis by Cornish-Fisher expansion

Authors: Ningning Zhang, Ke Zhu

Abstract: The conditional variance, skewness, and kurtosis play a central role in time series analysis. These three conditional moments (CMs) are often studied by some parametric models but with two big issues: the risk of model mis-specification and the instability of model estimation. To avoid the above two issues, this paper proposes a novel method to estimate these three CMs by the so-called quantiled C… ▽ More The conditional variance, skewness, and kurtosis play a central role in time series analysis. These three conditional moments (CMs) are often studied by some parametric models but with two big issues: the risk of model mis-specification and the instability of model estimation. To avoid the above two issues, this paper proposes a novel method to estimate these three CMs by the so-called quantiled CMs (QCMs). The QCM method first adopts the idea of Cornish-Fisher expansion to construct a linear regression model, based on $n$ different estimated conditional quantiles. Next, it computes the QCMs simply and simultaneously by using the ordinary least squares estimator of this regression model, without any prior estimation of the conditional mean. Under certain conditions, the QCMs are shown to be consistent with the convergence rate $n^{-1/2}$. Simulation studies indicate that the QCMs perform well under different scenarios of Cornish-Fisher expansion errors and quantile estimation errors. In the application, the study of QCMs for three exchange rates demonstrates the effectiveness of financial rescue plans during the COVID-19 pandemic outbreak, and suggests that the existing ``news impact curve'' functions for the conditional skewness and kurtosis may not be suitable. △ Less

Submitted 6 June, 2023; v1 submitted 13 February, 2023; originally announced February 2023.

Big portfolio selection by graph-based conditional moments method

Authors: Zhoufan Zhu, Ningning Zhang, Ke Zhu

Abstract: How to do big portfolio selection is very important but challenging for both researchers and practitioners. In this paper, we propose a new graph-based conditional moments (GRACE) method to do portfolio selection based on thousands of stocks or more. The GRACE method first learns the conditional quantiles and mean of stock returns via a factor-augmented temporal graph convolutional network, which… ▽ More How to do big portfolio selection is very important but challenging for both researchers and practitioners. In this paper, we propose a new graph-based conditional moments (GRACE) method to do portfolio selection based on thousands of stocks or more. The GRACE method first learns the conditional quantiles and mean of stock returns via a factor-augmented temporal graph convolutional network, which guides the learning procedure through a factor-hypergraph built by the set of stock-to-stock relations from the domain knowledge as well as the set of factor-to-stock relations from the asset pricing knowledge. Next, the GRACE method learns the conditional variance, skewness, and kurtosis of stock returns from the learned conditional quantiles by using the quantiled conditional moment (QCM) method. The QCM method is a supervised learning procedure to learn these conditional higher-order moments, so it largely overcomes the computational difficulty from the classical high-dimensional GARCH-type methods. Moreover, the QCM method allows the mis-specification in modeling conditional quantiles to some extent, due to its regression-based nature. Finally, the GRACE method uses the learned conditional mean, variance, skewness, and kurtosis to construct several performance measures, which are criteria to sort the stocks to proceed the portfolio selection in the well-known 10-decile framework. An application to NASDAQ and NYSE stock markets shows that the GRACE method performs much better than its competitors, particularly when the performance measures are comprised of conditional variance, skewness, and kurtosis. △ Less

Submitted 27 January, 2023; originally announced January 2023.

Comments: 35 pages

MSC Class: 62M10; 68T07; 91G10

Statistical inference for the logarithmic spatial heteroskedasticity model with exogenous variables

Authors: Bing Su, Fukang Zhu, Ke Zhu

Abstract: The spatial dependence in mean has been well studied by plenty of models in a large strand of literature, however, the investigation of spatial dependence in variance is lagging significantly behind. The existing models for the spatial dependence in variance are scarce, with neither probabilistic structure nor statistical inference procedure being explored. To circumvent this deficiency, this pape… ▽ More The spatial dependence in mean has been well studied by plenty of models in a large strand of literature, however, the investigation of spatial dependence in variance is lagging significantly behind. The existing models for the spatial dependence in variance are scarce, with neither probabilistic structure nor statistical inference procedure being explored. To circumvent this deficiency, this paper proposes a new generalized logarithmic spatial heteroscedasticity model with exogenous variables (denoted by the log-SHE model) to study the spatial dependence in variance. For the log-SHE model, its spatial near-epoch dependence (NED) property is investigated, and a systematic statistical inference procedure is provided, including the maximum likelihood and generalized method of moments estimators, the Wald, Lagrange multiplier and likelihood-ratio-type D tests for model parameter constraints, and the overidentification test for the model diagnostic checking. Using the tool of spatial NED, the asymptotics of all proposed estimators and tests are established under regular conditions. The usefulness of the proposed methodology is illustrated by simulation results and a real data example on the house selling price. △ Less

Submitted 16 January, 2023; originally announced January 2023.

Design-based theory for Lasso adjustment in randomized block experiments and rerandomized experiments

Authors: Ke Zhu, Hanzhong Liu, Yuehan Yang

Abstract: Blocking, a special case of rerandomization, is routinely implemented in the design stage of randomized experiments to balance the baseline covariates. This study proposes a regression adjustment method based on the least absolute shrinkage and selection operator (Lasso) to efficiently estimate the average treatment effect in randomized block experiments with high-dimensional covariates. We derive… ▽ More Blocking, a special case of rerandomization, is routinely implemented in the design stage of randomized experiments to balance the baseline covariates. This study proposes a regression adjustment method based on the least absolute shrinkage and selection operator (Lasso) to efficiently estimate the average treatment effect in randomized block experiments with high-dimensional covariates. We derive the asymptotic properties of the proposed estimator and outline the conditions under which this estimator is more efficient than the unadjusted one. We provide a conservative variance estimator to facilitate valid inferences. Our framework allows one treated or control unit in some blocks and heterogeneous propensity scores across blocks, thus including paired experiments and finely stratified experiments as special cases. We further accommodate rerandomized experiments and a combination of blocking and rerandomization. Moreover, our analysis allows both the number of blocks and block sizes to tend to infinity, as well as heterogeneous treatment effects across blocks without assuming a true outcome data-generating model. Simulation studies and two real-data analyses demonstrate the advantages of the proposed method. △ Less

Submitted 28 June, 2024; v1 submitted 23 September, 2021; originally announced September 2021.

Path-specific Effects Based on Information Accounts of Causality

Authors: Heyang Gong, Ke Zhu

Abstract: Path-specific effects in mediation analysis provide a useful tool for fairness analysis, which is mostly based on nested counterfactuals. However, the dictum ``no causation without manipulation'' implies that path-specific effects might be induced by certain interventions. This paper proposes a new path intervention inspired by information accounts of causality, and develops the corresponding inte… ▽ More Path-specific effects in mediation analysis provide a useful tool for fairness analysis, which is mostly based on nested counterfactuals. However, the dictum ``no causation without manipulation'' implies that path-specific effects might be induced by certain interventions. This paper proposes a new path intervention inspired by information accounts of causality, and develops the corresponding intervention diagrams and $π$-formula. Compared with the interventionist approach of Robins et al.(2020) based on nested counterfactuals, our proposed path intervention method explicitly describes the manipulation in structural causal model with a simple information transferring interpretation, and does not require the non-existence of recanting witness to identify path-specific effects. Hence, it could serve useful communications and theoretical focus for mediation analysis. △ Less

Submitted 6 June, 2021; originally announced June 2021.

Quantifying out-of-station waiting time in oversaturated urban metro systems

Authors: Kangli Zhu, Zhanhong Cheng, Jianjun Wu, Fuya Yuan, Lijun Sun

Abstract: Metro systems in megacities such as Beijing, Shenzhen and Guangzhou are under great passenger demand pressure. During peak hours, it is common to see oversaturated conditions (i.e., passenger demand exceeds network capacity), which bring significant operational risks and safety issues. A popular control intervention is to restrict the entering rate during peak hours by setting up out-of-station qu… ▽ More Metro systems in megacities such as Beijing, Shenzhen and Guangzhou are under great passenger demand pressure. During peak hours, it is common to see oversaturated conditions (i.e., passenger demand exceeds network capacity), which bring significant operational risks and safety issues. A popular control intervention is to restrict the entering rate during peak hours by setting up out-of-station queueing with crowd control barriers. The \textit{out-of-station waiting} can make up a substantial proportion of total travel time but is not well-studied in the literature. Accurate quantification of out-of-station waiting time is important to evaluating the social benefit and cost of service scheduling/optimization plans; however, out-of-station waiting time is difficult to estimate because it is not a part of smart card transactions. In this study, we propose an innovative method to estimate the out-of-station waiting time by leveraging the information from a small group of transfer passengers -- those who transfer from nearby bus routes to the metro station. Based on the estimated transfer time for this small group, we first infer the out-of-station waiting time for all passengers by developing a Gaussian Process regression with a Student-$t$ likelihood and then use the estimated out-of-station waiting time to build queueing diagrams. We apply our method to the Tiantongyuan North station of Beijing metro as a case study; our results show that the maximum out-of-station waiting time can reach 15 minutes, and the maximum queue length can be over 3000 passengers. Our results suggest that out-of-station waiting can cause significant travel costs and thus should be considered in analyzing transit performance, mode choice, and social benefits. To the best of our knowledge, this paper is the first quantitative study for out-of-station waiting time. △ Less

Submitted 1 June, 2021; originally announced June 2021.

Journal ref: Communications in Transportation Research (2022)

Pair-switching rerandomization

Authors: Ke Zhu, Hanzhong Liu

Abstract: Rerandomization discards assignments with covariates unbalanced in the treatment and control groups to improve estimation and inference efficiency. However, the acceptance-rejection sampling method used in rerandomization is computationally inefficient. As a result, it is time-consuming for rerandomization to draw numerous independent assignments, which are necessary for performing Fisher randomiz… ▽ More Rerandomization discards assignments with covariates unbalanced in the treatment and control groups to improve estimation and inference efficiency. However, the acceptance-rejection sampling method used in rerandomization is computationally inefficient. As a result, it is time-consuming for rerandomization to draw numerous independent assignments, which are necessary for performing Fisher randomization tests and constructing randomization-based confidence intervals. To address this problem, we propose a pair-switching rerandomization method to draw balanced assignments efficiently. We obtain the unbiasedness and variance reduction of the difference-in-means estimator and show that the Fisher randomization tests are valid under pair-switching rerandomization. Moreover, we propose an exact approach to invert Fisher randomization tests to confidence intervals, which is faster than the existing methods. In addition, our method is applicable to both non-sequentially and sequentially randomized experiments. We conduct comprehensive simulation studies to compare the finite-sample performance of the proposed method with that of classical rerandomization. Simulation results indicate that pair-switching rerandomization leads to comparable power of Fisher randomization tests and is 3--23 times faster than classical rerandomization. Finally, we apply the pair-switching rerandomization method to analyze two clinical trial datasets, both of which demonstrate the advantages of our method. △ Less

Submitted 25 June, 2022; v1 submitted 24 March, 2021; originally announced March 2021.

Confidence intervals for parameters in high-dimensional sparse vector autoregression

Authors: Ke Zhu, Hanzhong Liu

Abstract: Vector autoregression (VAR) models are widely used to analyze the interrelationship between multiple variables over time. Estimation and inference for the transition matrices of VAR models are crucial for practitioners to make decisions in fields such as economics and finance. However, when the number of variables is larger than the sample size, it remains a challenge to perform statistical infere… ▽ More Vector autoregression (VAR) models are widely used to analyze the interrelationship between multiple variables over time. Estimation and inference for the transition matrices of VAR models are crucial for practitioners to make decisions in fields such as economics and finance. However, when the number of variables is larger than the sample size, it remains a challenge to perform statistical inference of the model parameters. In this article, we propose the de-biased Lasso and two bootstrap de-biased Lasso methods to construct confidence intervals for the elements of the transition matrices of high-dimensional VAR models. We show that the proposed methods are asymptotically valid under appropriate sparsity and other regularity conditions. To implement our methods, we develop feasible and parallelizable algorithms, which save a large amount of computation required by the nodewise Lasso and bootstrap. A simulation study illustrates that our methods perform well in finite samples. Finally, we apply our methods to analyze the price data of stocks in the S&P 500 index in 2019. We find that some stocks, such as the largest producer of gold in the world, Newmont Corporation, have significant predictive power over the most stocks. △ Less

Submitted 20 September, 2020; originally announced September 2020.

Testing error distribution by kernelized Stein discrepancy in multivariate time series models

Authors: Donghang Luo, Ke Zhu, Huan Gong, Dong Li

Abstract: Knowing the error distribution is important in many multivariate time series applications. To alleviate the risk of error distribution mis-specification, testing methodologies are needed to detect whether the chosen error distribution is correct. However, the majority of the existing tests only deal with the multivariate normal distribution for some special multivariate time series models, and the… ▽ More Knowing the error distribution is important in many multivariate time series applications. To alleviate the risk of error distribution mis-specification, testing methodologies are needed to detect whether the chosen error distribution is correct. However, the majority of the existing tests only deal with the multivariate normal distribution for some special multivariate time series models, and they thus can not be used to testing for the often observed heavy-tailed and skewed error distributions in applications. In this paper, we construct a new consistent test for general multivariate time series models, based on the kernelized Stein discrepancy. To account for the estimation uncertainty and unobserved initial values, a bootstrap method is provided to calculate the critical values. Our new test is easy-to-implement for a large scope of multivariate error distributions, and its importance is illustrated by simulated and real data. △ Less

Submitted 3 August, 2020; originally announced August 2020.

MICK: A Meta-Learning Framework for Few-shot Relation Classification with Small Training Data

Authors: Xiaoqing Geng, Xiwen Chen, Kenny Q. Zhu, Libin Shen, Yinggong Zhao

Abstract: Few-shot relation classification seeks to classify incoming query instances after meeting only few support instances. This ability is gained by training with large amount of in-domain annotated data. In this paper, we tackle an even harder problem by further limiting the amount of data available at training time. We propose a few-shot learning framework for relation classification, which is partic… ▽ More Few-shot relation classification seeks to classify incoming query instances after meeting only few support instances. This ability is gained by training with large amount of in-domain annotated data. In this paper, we tackle an even harder problem by further limiting the amount of data available at training time. We propose a few-shot learning framework for relation classification, which is particularly powerful when the training data is very small. In this framework, models not only strive to classify query instances, but also seek underlying knowledge about the support instances to obtain better instance representations. The framework also includes a method for aggregating cross-domain knowledge into models by open-source task enrichment. Additionally, we construct a brand new dataset: the TinyRel-CM dataset, a few-shot relation classification dataset in health domain with purposely small training data and challenging relation classes. Experimental results demonstrate that our framework brings performance gains for most underlying classification models, outperforms the state-of-the-art results given small training data, and achieves competitive results with sufficiently large training data. △ Less

Submitted 14 December, 2020; v1 submitted 26 April, 2020; originally announced April 2020.

Journal ref: CIKM 2020: The 29th ACM International Conference on Information and Knowledge Management

Multi-frequency-band tests for white noise under heteroskedasticity

Authors: Mengya Liu, Fukan Zhu, Ke Zhu

Abstract: This paper proposes a new family of multi-frequency-band (MFB) tests for the white noise hypothesis by using the maximum overlap discrete wavelet packet transform (MODWPT). The MODWPT allows the variance of a process to be decomposed into the variance of its components on different equal-length frequency sub-bands, and the MFB tests then measure the distance between the MODWPT-based variance ratio… ▽ More This paper proposes a new family of multi-frequency-band (MFB) tests for the white noise hypothesis by using the maximum overlap discrete wavelet packet transform (MODWPT). The MODWPT allows the variance of a process to be decomposed into the variance of its components on different equal-length frequency sub-bands, and the MFB tests then measure the distance between the MODWPT-based variance ratio and its theoretical null value jointly over several frequency sub-bands. The resulting MFB tests have the chi-squared asymptotic null distributions under mild conditions, which allow the data to be heteroskedastic. The MFB tests are shown to have the desirable size and power performance by simulation studies, and their usefulness is further illustrated by two applications. △ Less

Submitted 20 April, 2020; originally announced April 2020.

Unpack Local Model Interpretation for GBDT

Authors: Wenjing Fang, Jun Zhou, Xiaolong Li, Kenny Q. Zhu

Abstract: A gradient boosting decision tree (GBDT), which aggregates a collection of single weak learners (i.e. decision trees), is widely used for data mining tasks. Because GBDT inherits the good performance from its ensemble essence, much attention has been drawn to the optimization of this model. With its popularization, an increasing need for model interpretation arises. Besides the commonly used featu… ▽ More A gradient boosting decision tree (GBDT), which aggregates a collection of single weak learners (i.e. decision trees), is widely used for data mining tasks. Because GBDT inherits the good performance from its ensemble essence, much attention has been drawn to the optimization of this model. With its popularization, an increasing need for model interpretation arises. Besides the commonly used feature importance as a global interpretation, feature contribution is a local measure that reveals the relationship between a specific instance and the related output. This work focuses on the local interpretation and proposes an unified computation mechanism to get the instance-level feature contributions for GBDT in any version. Practicality of this mechanism is validated by the listed experiments as well as applications in real industry scenarios. △ Less

Submitted 2 April, 2020; originally announced April 2020.

Comments: 12 pages, 5 figures

Adapted tree boosting for Transfer Learning

Authors: Wenjing Fang, Chaochao Chen, Bowen Song, Li Wang, Jun Zhou, Kenny Q. Zhu

Abstract: Secure online transaction is an essential task for e-commerce platforms. Alipay, one of the world's leading cashless payment platform, provides the payment service to both merchants and individual customers. The fraud detection models are built to protect the customers, but stronger demands are raised by the new scenes, which are lacking in training data and labels. The proposed model makes a diff… ▽ More Secure online transaction is an essential task for e-commerce platforms. Alipay, one of the world's leading cashless payment platform, provides the payment service to both merchants and individual customers. The fraud detection models are built to protect the customers, but stronger demands are raised by the new scenes, which are lacking in training data and labels. The proposed model makes a difference by utilizing the data under similar old scenes and the data under a new scene is treated as the target domain to be promoted. Inspired by this real case in Alipay, we view the problem as a transfer learning problem and design a set of revise strategies to transfer the source domain models to the target domain under the framework of gradient boosting tree models. This work provides an option for the cold-starting and data-sharing problems. △ Less

Submitted 2 April, 2020; v1 submitted 27 February, 2020; originally announced February 2020.

ACM Class: I.2.6

Hybrid quantile estimation for asymmetric power GARCH models

Authors: Guochang Wang, Ke Zhu, Guodong Li, Wai Keung Li

Abstract: Asymmetric power GARCH models have been widely used to study the higher order moments of financial returns, while their quantile estimation has been rarely investigated. This paper introduces a simple monotonic transformation on its conditional quantile function to make the quantile regression tractable. The asymptotic normality of the resulting quantile estimators is established under either stat… ▽ More Asymmetric power GARCH models have been widely used to study the higher order moments of financial returns, while their quantile estimation has been rarely investigated. This paper introduces a simple monotonic transformation on its conditional quantile function to make the quantile regression tractable. The asymptotic normality of the resulting quantile estimators is established under either stationarity or non-stationarity. Moreover, based on the estimation procedure, new tests for strict stationarity and asymmetry are also constructed. This is the first try of the quantile estimation for non-stationary ARCH-type models in the literature. The usefulness of the proposed methodology is illustrated by simulation results and real data analysis. △ Less

Submitted 21 November, 2019; originally announced November 2019.

Adaptive inference for a semiparametric generalized autoregressive conditional heteroskedasticity model

Authors: Feiyu Jiang, Dong Li, Ke Zhu

Abstract: This paper considers a semiparametric generalized autoregressive conditional heteroskedasticity (S-GARCH) model. For this model, we first estimate the time-varying long run component for unconditional variance by the kernel estimator, and then estimate the non-time-varying parameters in GARCH-type short run component by the quasi maximum likelihood estimator (QMLE). We show that the QMLE is asympt… ▽ More This paper considers a semiparametric generalized autoregressive conditional heteroskedasticity (S-GARCH) model. For this model, we first estimate the time-varying long run component for unconditional variance by the kernel estimator, and then estimate the non-time-varying parameters in GARCH-type short run component by the quasi maximum likelihood estimator (QMLE). We show that the QMLE is asymptotically normal with the parametric convergence rate. Next, we construct a Lagrange multiplier test for linear parameter constraint and a portmanteau test for model checking, and obtain their asymptotic null distributions. Our entire statistical inference procedure works for the non-stationary data with two important features: first, our QMLE and two tests are adaptive to the unknown form of the long run component; second, our QMLE and two tests share the same efficiency and testing power as those in variance targeting method when the S-GARCH model is stationary. △ Less

Submitted 2 October, 2020; v1 submitted 9 July, 2019; originally announced July 2019.

Data-Driven Machine Learning Techniques for Self-healing in Cellular Wireless Networks: Challenges and Solutions

Authors: Tao Zhang, Kun Zhu, Ekram Hossain

Abstract: For enabling automatic deployment and management of cellular networks, the concept of self-organizing network (SON) was introduced. SON capabilities can enhance network performance, improve service quality, and reduce operational and capital expenditure (OPEX/CAPEX). As an important component in SON, self-healing is defined as a network paradigm where the faults of target networks are mitigated or… ▽ More For enabling automatic deployment and management of cellular networks, the concept of self-organizing network (SON) was introduced. SON capabilities can enhance network performance, improve service quality, and reduce operational and capital expenditure (OPEX/CAPEX). As an important component in SON, self-healing is defined as a network paradigm where the faults of target networks are mitigated or recovered by automatically triggering a series of actions such as detection, diagnosis and compensation. Data-driven machine learning has been recognized as a powerful tool to bring intelligence into network and to realize self-healing. However, there are major challenges for practical applications of machine learning techniques for self-healing. In this article, we first classify these challenges into five categories: 1) data imbalance, 2) data insufficiency, 3) cost insensitivity, 4) non-real-time response, and 5) multi-source data fusion. Then we provide potential technical solutions to address these challenges. Furthermore, a case study of cost-sensitive fault detection with imbalanced data is provided to illustrate the feasibility and effectiveness of the suggested solutions. △ Less

Submitted 14 June, 2019; originally announced June 2019.

Non-standard inference for augmented double autoregressive models with null volatility coefficients

Authors: Feiyu Jiang, Dong Li, Ke Zhu

Abstract: This paper considers an augmented double autoregressive (DAR) model, which allows null volatility coefficients to circumvent the over-parameterization problem in the DAR model. Since the volatility coefficients might be on the boundary, the statistical inference methods based on the Gaussian quasi-maximum likelihood estimation (GQMLE) become non-standard, and their asymptotics require the data to… ▽ More This paper considers an augmented double autoregressive (DAR) model, which allows null volatility coefficients to circumvent the over-parameterization problem in the DAR model. Since the volatility coefficients might be on the boundary, the statistical inference methods based on the Gaussian quasi-maximum likelihood estimation (GQMLE) become non-standard, and their asymptotics require the data to have a finite sixth moment, which narrows applicable scope in studying heavy-tailed data. To overcome this deficiency, this paper develops a systematic statistical inference procedure based on the self-weighted GQMLE for the augmented DAR model. Except for the Lagrange multiplier test statistic, the Wald, quasi-likelihood ratio and portmanteau test statistics are all shown to have non-standard asymptotics. The entire procedure is valid as long as the data is stationary, and its usefulness is illustrated by simulation studies and one real example. △ Less

Submitted 5 May, 2019; originally announced May 2019.

Time series models for realized covariance matrices based on the matrix-F distribution

Authors: Jiayuan Zhou, Feiyu Jiang, Ke Zhu, Wai Keung Li

Abstract: We propose a new Conditional BEKK matrix-F (CBF) model for the time-varying realized covariance (RCOV) matrices. This CBF model is capable of capturing heavy-tailed RCOV, which is an important stylized fact but could not be handled adequately by the Wishart-based models. To further mimic the long memory feature of the RCOV, a special CBF model with the conditional heterogeneous autoregressive (HAR… ▽ More We propose a new Conditional BEKK matrix-F (CBF) model for the time-varying realized covariance (RCOV) matrices. This CBF model is capable of capturing heavy-tailed RCOV, which is an important stylized fact but could not be handled adequately by the Wishart-based models. To further mimic the long memory feature of the RCOV, a special CBF model with the conditional heterogeneous autoregressive (HAR) structure is introduced. Moreover, we give a systematical study on the probabilistic properties and statistical inferences of the CBF model, including exploring its stationarity, establishing the asymptotics of its maximum likelihood estimator, and giving some new inner-product-based tests for its model checking. In order to handle a large dimensional RCOV matrix, we construct two reduced CBF models -- the variance-target CBF model (for moderate but fixed dimensional RCOV matrix) and the factor CBF model (for high dimensional RCOV matrix). For both reduced models, the asymptotic theory of the estimated parameters is derived. The importance of our entire methodology is illustrated by simulation results and two real examples. △ Less

Submitted 9 July, 2020; v1 submitted 26 March, 2019; originally announced March 2019.

Understanding Childhood Vulnerability in The City of Surrey

Authors: Cody Griffith, Varoon Mathur, Catherine Lin, Kevin Zhu

Abstract: Understanding the community conditions that best support universal access and improved childhood outcomes allows ultimately to improve decision-making in the areas of planning and investment across the early stages of childhood development. Here we describe two different data-driven approaches to visualizing the lived experiences of children throughout the City of Surrey, combining data derived fr… ▽ More Understanding the community conditions that best support universal access and improved childhood outcomes allows ultimately to improve decision-making in the areas of planning and investment across the early stages of childhood development. Here we describe two different data-driven approaches to visualizing the lived experiences of children throughout the City of Surrey, combining data derived from both public and private sources. In one approach, we find specifically that the Early Development Instrument measuring childhood vulnerabilities across varying domains can be used to cluster neighborhoods, and that census variables can help explain similarities between neighborhoods within these clusters. In our second approach, we use program registration data from the City of Surrey's Community and Recreation Services Division. We also find a critical age of entry and exit for each program related to early childhood development and beyond, and find that certain neighborhoods and recreational programs have larger retention rates than others. This report details the journey of using data to tell the story of these neighborhoods, and provides a lens to which community initiatives can be strategically crafted through their use. △ Less

Submitted 25 March, 2019; originally announced March 2019.

New HSIC-based tests for independence between two stationary multivariate time series

Authors: Guochang Wang, Wai Keung Li, Ke Zhu

Abstract: This paper proposes some novel one-sided omnibus tests for independence between two multivariate stationary time series. These new tests apply the Hilbert-Schmidt independence criterion (HSIC) to test the independence between the innovations of both time series. Under regular conditions, the limiting null distributions of our HSIC-based tests are established. Next, our HSIC-based tests are shown t… ▽ More This paper proposes some novel one-sided omnibus tests for independence between two multivariate stationary time series. These new tests apply the Hilbert-Schmidt independence criterion (HSIC) to test the independence between the innovations of both time series. Under regular conditions, the limiting null distributions of our HSIC-based tests are established. Next, our HSIC-based tests are shown to be consistent. Moreover, a residual bootstrap method is used to obtain the critical values for our HSIC-based tests, and its validity is justified. Compared with the existing cross-correlation-based tests for linear dependence, our tests examine the general (including both linear and non-linear) dependence to give investigators more complete information on the causal relationship between two multivariate time series. The merits of our tests are illustrated by some simulation results and a real example. △ Less

Submitted 25 April, 2018; originally announced April 2018.

Statistical inference for autoregressive models under heteroscedasticity of unknown form

Authors: Ke Zhu

Abstract: This paper provides an entire inference procedure for the autoregressive model under (conditional) heteroscedasticity of unknown form with a finite variance. We first establish the asymptotic normality of the weighted least absolute deviations estimator (LADE) for the model. Second, we develop the random weighting (RW) method to estimate its asymptotic covariance matrix, leading to the implementat… ▽ More This paper provides an entire inference procedure for the autoregressive model under (conditional) heteroscedasticity of unknown form with a finite variance. We first establish the asymptotic normality of the weighted least absolute deviations estimator (LADE) for the model. Second, we develop the random weighting (RW) method to estimate its asymptotic covariance matrix, leading to the implementation of the Wald test. Third, we construct a portmanteau test for model checking, and use the RW method to obtain its critical values. As a special weighted LADE, the feasible adaptive LADE (ALADE) is proposed and proved to have the same efficiency as its infeasible counterpart. The importance of our entire methodology based on the feasible ALADE is illustrated by simulation results and the real data analysis on three U.S. economic data sets. △ Less

Submitted 8 August, 2018; v1 submitted 6 April, 2018; originally announced April 2018.

CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning

Authors: Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, Matthew P. Lungren, Andrew Y. Ng

Abstract: We develop an algorithm that can detect pneumonia from chest X-rays at a level exceeding practicing radiologists. Our algorithm, CheXNet, is a 121-layer convolutional neural network trained on ChestX-ray14, currently the largest publicly available chest X-ray dataset, containing over 100,000 frontal-view X-ray images with 14 diseases. Four practicing academic radiologists annotate a test set, on w… ▽ More We develop an algorithm that can detect pneumonia from chest X-rays at a level exceeding practicing radiologists. Our algorithm, CheXNet, is a 121-layer convolutional neural network trained on ChestX-ray14, currently the largest publicly available chest X-ray dataset, containing over 100,000 frontal-view X-ray images with 14 diseases. Four practicing academic radiologists annotate a test set, on which we compare the performance of CheXNet to that of radiologists. We find that CheXNet exceeds average radiologist performance on the F1 metric. We extend CheXNet to detect all 14 diseases in ChestX-ray14 and achieve state of the art results on all 14 diseases. △ Less

Submitted 25 December, 2017; v1 submitted 14 November, 2017; originally announced November 2017.

Practical Algorithms for Learning Near-Isometric Linear Embeddings

Authors: Jerry Luo, Kayla Shapiro, Hao-Jun Michael Shi, Qi Yang, Kan Zhu

Abstract: We propose two practical non-convex approaches for learning near-isometric, linear embeddings of finite sets of data points. Given a set of training points $\mathcal{X}$, we consider the secant set $S(\mathcal{X})$ that consists of all pairwise difference vectors of $\mathcal{X}$, normalized to lie on the unit sphere. The problem can be formulated as finding a symmetric and positive semi-definite… ▽ More We propose two practical non-convex approaches for learning near-isometric, linear embeddings of finite sets of data points. Given a set of training points $\mathcal{X}$, we consider the secant set $S(\mathcal{X})$ that consists of all pairwise difference vectors of $\mathcal{X}$, normalized to lie on the unit sphere. The problem can be formulated as finding a symmetric and positive semi-definite matrix $\boldsymbolΨ$ that preserves the norms of all the vectors in $S(\mathcal{X})$ up to a distortion parameter $δ$. Motivated by non-negative matrix factorization, we reformulate our problem into a Frobenius norm minimization problem, which is solved by the Alternating Direction Method of Multipliers (ADMM) and develop an algorithm, FroMax. Another method solves for a projection matrix $\boldsymbolΨ$ by minimizing the restricted isometry property (RIP) directly over the set of symmetric, postive semi-definite matrices. Applying ADMM and a Moreau decomposition on a proximal mapping, we develop another algorithm, NILE-Pro, for dimensionality reduction. FroMax is shown to converge faster for smaller $δ$ while NILE-Pro converges faster for larger $δ$. Both non-convex approaches are then empirically demonstrated to be more computationally efficient than prior convex approaches for a number of applications in machine learning and signal processing. △ Less

Submitted 22 April, 2016; v1 submitted 1 January, 2016; originally announced January 2016.

MSC Class: 90C90

Collaborative Filtering with Information-Rich and Information-Sparse Entities

Authors: Kai Zhu, Rui Wu, Lei Ying, R. Srikant

Abstract: In this paper, we consider a popular model for collaborative filtering in recommender systems where some users of a website rate some items, such as movies, and the goal is to recover the ratings of some or all of the unrated items of each user. In particular, we consider both the clustering model, where only users (or items) are clustered, and the co-clustering model, where both users and items a… ▽ More In this paper, we consider a popular model for collaborative filtering in recommender systems where some users of a website rate some items, such as movies, and the goal is to recover the ratings of some or all of the unrated items of each user. In particular, we consider both the clustering model, where only users (or items) are clustered, and the co-clustering model, where both users and items are clustered, and further, we assume that some users rate many items (information-rich users) and some users rate only a few items (information-sparse users). When users (or items) are clustered, our algorithm can recover the rating matrix with $ω(MK \log M)$ noisy entries while $MK$ entries are necessary, where $K$ is the number of clusters and $M$ is the number of items. In the case of co-clustering, we prove that $K^2$ entries are necessary for recovering the rating matrix, and our algorithm achieves this lower bound within a logarithmic factor when $K$ is sufficiently large. We compare our algorithms with a well-known algorithms called alternating minimization (AM), and a similarity score-based algorithm known as the popularity-among-friends (PAF) algorithm by applying all three to the MovieLens and Netflix data sets. Our co-clustering algorithm and AM have similar overall error rates when recovering the rating matrix, both of which are lower than the error rate under PAF. But more importantly, the error rate of our co-clustering algorithm is significantly lower than AM and PAF in the scenarios of interest in recommender systems: when recommending a few items to each user or when recommending items to users who only rated a few items (these users are the majority of the total user population). The performance difference increases even more when noise is added to the datasets. △ Less

Submitted 6 March, 2014; originally announced March 2014.

Jointly Clustering Rows and Columns of Binary Matrices: Algorithms and Trade-offs

Authors: Jiaming Xu, Rui Wu, Kai Zhu, Bruce Hajek, R. Srikant, Lei Ying

Abstract: In standard clustering problems, data points are represented by vectors, and by stacking them together, one forms a data matrix with row or column cluster structure. In this paper, we consider a class of binary matrices, arising in many applications, which exhibit both row and column cluster structure, and our goal is to exactly recover the underlying row and column clusters by observing only a sm… ▽ More In standard clustering problems, data points are represented by vectors, and by stacking them together, one forms a data matrix with row or column cluster structure. In this paper, we consider a class of binary matrices, arising in many applications, which exhibit both row and column cluster structure, and our goal is to exactly recover the underlying row and column clusters by observing only a small fraction of noisy entries. We first derive a lower bound on the minimum number of observations needed for exact cluster recovery. Then, we propose three algorithms with different running time and compare the number of observations needed by them for successful cluster recovery. Our analytical results show smooth time-data trade-offs: one can gradually reduce the computational complexity when increasingly more observations are available. △ Less

Submitted 4 February, 2014; v1 submitted 1 October, 2013; originally announced October 2013.

LAGE: A Java Framework to reconstruct Gene Regulatory Networks from Large-Scale Continues Expression Data

Authors: Yang Lu, Mengying Wang, Kenny Q. Zhu, Bo Yuan

Abstract: LAGE is a systematic framework developed in Java. The motivation of LAGE is to provide a scalable and parallel solution to reconstruct Gene Regulatory Networks (GRNs) from continuous gene expression data for very large amount of genes. The basic idea of our framework is motivated by the philosophy of divideand-conquer. Specifically, LAGE recursively partitions genes into multiple overlapping commu… ▽ More LAGE is a systematic framework developed in Java. The motivation of LAGE is to provide a scalable and parallel solution to reconstruct Gene Regulatory Networks (GRNs) from continuous gene expression data for very large amount of genes. The basic idea of our framework is motivated by the philosophy of divideand-conquer. Specifically, LAGE recursively partitions genes into multiple overlapping communities with much smaller sizes, learns intra-community GRNs respectively before merge them altogether. Besides, the complete information of overlapping communities serves as the byproduct, which could be used to mine meaningful functional modules in biological networks. △ Less

Submitted 9 November, 2012; originally announced November 2012.

Comments: 2 pages