Showing 1–50 of 194 results for author: Lee, C

Robust Causal Inference for EHR-based Studies of Point Exposures with Missingness in Eligibility Criteria

Authors: Luke Benz, Rajarshi Mukherjee, Rui Wang, David Arterburn, Heidi Fischer, Catherine Lee, Susan M. Shortreed, Sebastien Haneuse, Alexander W. Levis

Abstract: Missingness in variables that define study eligibility criteria is a seldom addressed challenge in electronic health record (EHR)-based settings. It is typically the case that patients with incomplete eligibility information are excluded from analysis without consideration of (implicit) assumptions that are being made, leaving study conclusions subject to potential selection bias. In an effort to… ▽ More Missingness in variables that define study eligibility criteria is a seldom addressed challenge in electronic health record (EHR)-based settings. It is typically the case that patients with incomplete eligibility information are excluded from analysis without consideration of (implicit) assumptions that are being made, leaving study conclusions subject to potential selection bias. In an effort to ascertain eligibility for more patients, researchers may look back further in time prior to study baseline, and in using outdated values of eligibility-defining covariates may inappropriately be including individuals who, unbeknownst to the researcher, fail to meet eligibility at baseline. To the best of our knowledge, however, very little work has been done to mitigate these concerns. We propose a robust and efficient estimator of the causal average treatment effect on the treated, defined in the study eligible population, in cohort studies where eligibility-defining covariates are missing at random. The approach facilitates the use of flexible machine-learning strategies for component nuisance functions while maintaining appropriate convergence rates for valid asymptotic inference. EHR data from Kaiser Permanente are used as motivation as well as a basis for extensive simulations that verify robustness properties under various degrees of model misspecification. The data are also used to demonstrate the use of the method to analyze differences between two common bariatric surgical interventions for long-term weight and glycemic outcomes among a cohort of severely obese patients with type II diabetes mellitus. △ Less

Submitted 22 April, 2025; originally announced April 2025.

Scalable and robust regression models for continuous proportional data

Authors: Changwoo J. Lee, Benjamin K. Dahl, Otso Ovaskainen, David B. Dunson

Abstract: Beta regression is used routinely for continuous proportional data, but it often encounters practical issues such as a lack of robustness of regression parameter estimates to misspecification of the beta distribution. We develop an improved class of generalized linear models starting with the continuous binomial (cobin) distribution and further extending to dispersion mixtures of cobin distributio… ▽ More Beta regression is used routinely for continuous proportional data, but it often encounters practical issues such as a lack of robustness of regression parameter estimates to misspecification of the beta distribution. We develop an improved class of generalized linear models starting with the continuous binomial (cobin) distribution and further extending to dispersion mixtures of cobin distributions (micobin). The proposed cobin regression and micobin regression models have attractive robustness, computation, and flexibility properties. A key innovation is the Kolmogorov-Gamma data augmentation scheme, which facilitates Gibbs sampling for Bayesian computation, including in hierarchical cases involving nested, longitudinal, or spatial data. We demonstrate robustness, ability to handle responses exactly at the boundary (0 or 1), and computational efficiency relative to beta regression in simulation experiments and through analysis of the benthic macroinvertebrate multimetric index of US lakes using lake watershed covariates. △ Less

Submitted 21 April, 2025; originally announced April 2025.

On Learning Parallel Pancakes with Mostly Uniform Weights

Authors: Ilias Diakonikolas, Daniel M. Kane, Sushrut Karmalkar, Jasper C. H. Lee, Thanasis Pittas

Abstract: We study the complexity of learning $k$-mixtures of Gaussians ($k$-GMMs) on $\mathbb{R}^d$. This task is known to have complexity $d^{Ω(k)}$ in full generality. To circumvent this exponential lower bound on the number of components, research has focused on learning families of GMMs satisfying additional structural properties. A natural assumption posits that the component weights are not exponenti… ▽ More We study the complexity of learning $k$-mixtures of Gaussians ($k$-GMMs) on $\mathbb{R}^d$. This task is known to have complexity $d^{Ω(k)}$ in full generality. To circumvent this exponential lower bound on the number of components, research has focused on learning families of GMMs satisfying additional structural properties. A natural assumption posits that the component weights are not exponentially small and that the components have the same unknown covariance. Recent work gave a $d^{O(\log(1/w_{\min}))}$-time algorithm for this class of GMMs, where $w_{\min}$ is the minimum weight. Our first main result is a Statistical Query (SQ) lower bound showing that this quasi-polynomial upper bound is essentially best possible, even for the special case of uniform weights. Specifically, we show that it is SQ-hard to distinguish between such a mixture and the standard Gaussian. We further explore how the distribution of weights affects the complexity of this task. Our second main result is a quasi-polynomial upper bound for the aforementioned testing task when most of the weights are uniform while a small fraction of the weights are potentially arbitrary. △ Less

Submitted 21 April, 2025; originally announced April 2025.

Mean Independent Component Analysis for Multivariate Time Series

Authors: Chung Eun Lee, Zeda Li

Abstract: In this article, we introduce the mean independent component analysis for multivariate time series to reduce the parameter space. In particular, we seek for a contemporaneous linear transformation that detects univariate mean independent components so that each component can be modeled separately. The mean independent component analysis is flexible in the sense that no parametric model or distribu… ▽ More In this article, we introduce the mean independent component analysis for multivariate time series to reduce the parameter space. In particular, we seek for a contemporaneous linear transformation that detects univariate mean independent components so that each component can be modeled separately. The mean independent component analysis is flexible in the sense that no parametric model or distributional assumptions are made. We propose a unified framework to estimate the mean independent components from a data with a fixed dimension or a diverging dimension. We estimate the mean independent components by the martingale difference divergence so that the mean dependence across components and across time is minimized. The approach is extended to the group mean independent component analysis by imposing a group structure on the mean independent components. We further introduce a method to identify the group structure when it is unknown. The consistency of both proposed methods are established. Extensive simulations and a real data illustration for community mobility is provided to demonstrate the efficacy of our method. △ Less

Submitted 16 April, 2025; originally announced April 2025.

A Scorecard Model Using Survival Analysis Framework

Authors: Cheng Lee, Hsi Lee

Abstract: Credit risk assessment is a crucial aspect of financial decision-making, enabling institutions to predict the likelihood of default and make informed lending choices. Two prominent methodologies in risk modeling are logistic regression and survival analysis. Logistic regression is widely used for creating scorecard models due to its simplicity, interpretability, and effectiveness in estimating the… ▽ More Credit risk assessment is a crucial aspect of financial decision-making, enabling institutions to predict the likelihood of default and make informed lending choices. Two prominent methodologies in risk modeling are logistic regression and survival analysis. Logistic regression is widely used for creating scorecard models due to its simplicity, interpretability, and effectiveness in estimating the probability of binary outcomes, such as default versus non-default. On the other hand, survival analysis, particularly the hazard rate framework, offers insights into the timing of events, such as the time until default. By integrating logistic regression with survival analysis, traditional scorecard models can be enhanced to account not only for the probability of default but also for the dynamics of default over time. This combined approach provides a comprehensive view of credit risk, empowering institutions to manage risk proactively and tailor strategies to individual borrower profiles. In this article, the process of developing a scorecard model using logistic regression and augmenting data with survival analysis techniques to incorporate time-varying risk factors are presented. The process includes data preparation, model construction, evaluation metrics, and model implementation. △ Less

Submitted 6 March, 2025; originally announced March 2025.

A Generalized Theory of Mixup for Structure-Preserving Synthetic Data

Authors: Chungpa Lee, Jongho Im, Joseph H. T. Kim

Abstract: Mixup is a widely adopted data augmentation technique known for enhancing the generalization of machine learning models by interpolating between data points. Despite its success and popularity, limited attention has been given to understanding the statistical properties of the synthetic data it generates. In this paper, we delve into the theoretical underpinnings of mixup, specifically its effects… ▽ More Mixup is a widely adopted data augmentation technique known for enhancing the generalization of machine learning models by interpolating between data points. Despite its success and popularity, limited attention has been given to understanding the statistical properties of the synthetic data it generates. In this paper, we delve into the theoretical underpinnings of mixup, specifically its effects on the statistical structure of synthesized data. We demonstrate that while mixup improves model performance, it can distort key statistical properties such as variance, potentially leading to unintended consequences in data synthesis. To address this, we propose a novel mixup method that incorporates a generalized and flexible weighting scheme, better preserving the original data's structure. Through theoretical developments, we provide conditions under which our proposed method maintains the (co)variance and distributional properties of the original dataset. Numerical experiments confirm that the new approach not only preserves the statistical characteristics of the original data but also sustains model performance across repeated synthesis, alleviating concerns of model collapse identified in previous research. △ Less

Submitted 3 March, 2025; originally announced March 2025.

Journal ref: Proceedings of the 28th International Conference on Artificial Intelligence and Statistics (AISTATS) 2025

CMHSU: An R Statistical Software Package to Detect Mental Health Status, Substance Use Status, and their Concurrent Status in the North American Healthcare Administrative Databases

Authors: Mohsen Soltanifar, Chel Hee Lee

Abstract: The concept of concurrent mental health and substance use (MHSU) and its detection in patients has garnered growing interest among psychiatrists and healthcare policymakers over the past four decades. Researchers have proposed various diagnostic methods, including the Data-Driven Diagnostic Method (DDDM), for the identification of MHSU. However, the absence of a standalone statistical software pac… ▽ More The concept of concurrent mental health and substance use (MHSU) and its detection in patients has garnered growing interest among psychiatrists and healthcare policymakers over the past four decades. Researchers have proposed various diagnostic methods, including the Data-Driven Diagnostic Method (DDDM), for the identification of MHSU. However, the absence of a standalone statistical software package to facilitate DDDM for large healthcare administrative databases has remained a significant gap. This paper introduces the R statistical software package CMHSU, available on the Comprehensive R Archive Network (CRAN), for the diagnosis of mental health (MH), substance use (SU), and their concurrent status (MHSU). The package implements DDDM using hospital and medical service physician visit counts along with maximum time span parameters for MH, SU, and MHSU diagnoses. A working example using a simulated real-world dataset is presented to examine various analytical aspects, including three key dimensions of MHSU detection based on the DDDM framework, as well as temporal analysis to demonstrate the package's application for healthcare policymakers. Additionally, the limitations of the CMHSU package and potential directions for its future extension are discussed. △ Less

Submitted 6 April, 2025; v1 submitted 10 January, 2025; originally announced January 2025.

Comments: 21 pages; 7 figures; version 4.0

Report number: 6(2), 50

Journal ref: Psychiatry International, 2025

DisSim-FinBERT: Text Simplification for Core Message Extraction in Complex Financial Texts

Authors: Wonseong Kim, Christina Niklaus, Choong Lyol Lee, Siegfried Handschuh

Abstract: This study proposes DisSim-FinBERT, a novel framework that integrates Discourse Simplification (DisSim) with Aspect-Based Sentiment Analysis (ABSA) to enhance sentiment prediction in complex financial texts. By simplifying intricate documents such as Federal Open Market Committee (FOMC) minutes, DisSim improves the precision of aspect identification, resulting in sentiment predictions that align m… ▽ More This study proposes DisSim-FinBERT, a novel framework that integrates Discourse Simplification (DisSim) with Aspect-Based Sentiment Analysis (ABSA) to enhance sentiment prediction in complex financial texts. By simplifying intricate documents such as Federal Open Market Committee (FOMC) minutes, DisSim improves the precision of aspect identification, resulting in sentiment predictions that align more closely with economic events. The model preserves the original informational content and captures the inherent volatility of financial language, offering a more nuanced and accurate interpretation of long-form financial communications. This approach provides a practical tool for policymakers and analysts aiming to extract actionable insights from central bank narratives and other detailed economic documents. △ Less

Submitted 26 March, 2025; v1 submitted 8 January, 2025; originally announced January 2025.

Comments: 28 pages, 5 figures, 2 tables

Marginally interpretable spatial logistic regression with bridge processes

Authors: Changwoo J. Lee, David B. Dunson

Abstract: In including random effects to account for dependent observations, the odds ratio interpretation of logistic regression coefficients is changed from population-averaged to subject-specific. This is unappealing in many applications, motivating a rich literature on methods that maintain the marginal logistic regression structure without random effects, such as generalized estimating equations. Howev… ▽ More In including random effects to account for dependent observations, the odds ratio interpretation of logistic regression coefficients is changed from population-averaged to subject-specific. This is unappealing in many applications, motivating a rich literature on methods that maintain the marginal logistic regression structure without random effects, such as generalized estimating equations. However, for spatial data, random effect approaches are appealing in providing a full probabilistic characterization of the data that can be used for prediction. We propose a new class of spatial logistic regression models that maintain both population-averaged and subject-specific interpretations through a novel class of bridge processes for spatial random effects. These processes are shown to have appealing computational and theoretical properties, including a scale mixture of normal representation. The new methodology is illustrated with simulations and an analysis of childhood malaria prevalence data in the Gambia. △ Less

Submitted 28 February, 2025; v1 submitted 5 December, 2024; originally announced December 2024.

Conditional Extremes with Graphical Models

Authors: Aiden Farrell, Emma F. Eastoe, Clement Lee

Abstract: Multivariate extreme value analysis quantifies the probability and magnitude of joint extreme events. River discharges from the upper Danube River basin provide a challenging dataset for such analysis because the data, which is measured on a spatial network, exhibits both asymptotic dependence and asymptotic independence. To account for both features, we extend the conditional multivariate extreme… ▽ More Multivariate extreme value analysis quantifies the probability and magnitude of joint extreme events. River discharges from the upper Danube River basin provide a challenging dataset for such analysis because the data, which is measured on a spatial network, exhibits both asymptotic dependence and asymptotic independence. To account for both features, we extend the conditional multivariate extreme value model (CMEVM) with a new approach for the residual distribution. This allows sparse (graphical) dependence structures and fully parametric prediction. Our approach fills a current gap in statistical methodology by extending graphical extremes models to asymptotically independent random variables. Further, the model can be used to learn the graphical dependence structure when it is unknown a priori. To support inference in high dimensions, we propose a stepwise inference procedure that is computationally efficient and loses no information or predictive power. We show our method is flexible and accurately captures the extremal dependence for the upper Danube River basin discharges. △ Less

Submitted 11 April, 2025; v1 submitted 25 November, 2024; originally announced November 2024.

Bounded Rationality in Central Bank Communication

Authors: Wonseong Kim, Choong Lyol Lee

Abstract: This study explores the influence of FOMC sentiment on market expectations, focusing on cognitive differences between experts and non-experts. Using sentiment analysis of FOMC minutes, we integrate these insights into a bounded rationality model to examine the impact on inflation expectations. Results show that experts form more conservative expectations, anticipating FOMC stabilization actions, w… ▽ More This study explores the influence of FOMC sentiment on market expectations, focusing on cognitive differences between experts and non-experts. Using sentiment analysis of FOMC minutes, we integrate these insights into a bounded rationality model to examine the impact on inflation expectations. Results show that experts form more conservative expectations, anticipating FOMC stabilization actions, while non-experts react more directly to inflation concerns. A lead-lag analysis indicates that institutions adjust faster, though the gap with individual investors narrows in the short term. These findings highlight the need for tailored communication strategies to better align public expectations with policy goals. △ Less

Submitted 6 November, 2024; originally announced November 2024.

Comments: 72 pages, 5 figures, 8 tables

BLAST: Block-Level Adaptive Structured Matrices for Efficient Deep Neural Network Inference

Authors: Changwoo Lee, Soo Min Kwon, Qing Qu, Hun-Seok Kim

Abstract: Large-scale foundation models have demonstrated exceptional performance in language and vision tasks. However, the numerous dense matrix-vector operations involved in these large networks pose significant computational challenges during inference. To address these challenges, we introduce the Block-Level Adaptive STructured (BLAST) matrix, designed to learn and leverage efficient structures preval… ▽ More Large-scale foundation models have demonstrated exceptional performance in language and vision tasks. However, the numerous dense matrix-vector operations involved in these large networks pose significant computational challenges during inference. To address these challenges, we introduce the Block-Level Adaptive STructured (BLAST) matrix, designed to learn and leverage efficient structures prevalent in the weight matrices of linear layers within deep learning models. Compared to existing structured matrices, the BLAST matrix offers substantial flexibility, as it can represent various types of structures that are either learned from data or computed from pre-existing weight matrices. We demonstrate the efficiency of using the BLAST matrix for compressing both language and vision tasks, showing that (i) for medium-sized models such as ViT and GPT-2, training with BLAST weights boosts performance while reducing complexity by 70% and 40%, respectively; and (ii) for large foundation models such as Llama-7B and DiT-XL, the BLAST matrix achieves a 2x compression while exhibiting the lowest performance degradation among all tested structured matrices. Our code is available at https://github.com/changwoolee/BLAST. △ Less

Submitted 29 October, 2024; v1 submitted 28 October, 2024; originally announced October 2024.

Nonparametric Causal Survival Analysis with Clustered Interference

Authors: Chanhwa Lee, Donglin Zeng, Michael Emch, John D. Clemens, Michael G. Hudgens

Abstract: Inferring treatment effects on a survival time outcome based on data from an observational study is challenging due to the presence of censoring and possible confounding. An additional challenge occurs when a unit's treatment affects the outcome of other units, i.e., there is interference. In some settings, units may be grouped into clusters such that it is reasonable to assume interference only o… ▽ More Inferring treatment effects on a survival time outcome based on data from an observational study is challenging due to the presence of censoring and possible confounding. An additional challenge occurs when a unit's treatment affects the outcome of other units, i.e., there is interference. In some settings, units may be grouped into clusters such that it is reasonable to assume interference only occurs within clusters, i.e., there is clustered interference. In this paper, methods are developed which can accommodate confounding, censored outcomes, and clustered interference. The approach avoids parametric assumptions and permits inference about counterfactual scenarios corresponding to any stochastic policy which modifies the propensity score distribution, and thus may have application across diverse settings. The proposed nonparametric sample splitting estimators allow for flexible data-adaptive estimation of nuisance functions and are consistent and asymptotically normal with parametric convergence rates. Simulation studies demonstrate the finite sample performance of the proposed estimators, and the methods are applied to a cholera vaccine study in Bangladesh. △ Less

Submitted 19 September, 2024; originally announced September 2024.

Variable selection in the joint frailty model of recurrent and terminal events using Broken Adaptive Ridge regression

Authors: Christian Chan, Fatemeh Mahmoudi, Chel Hee Lee, Quan Long, Xuewen Lu

Abstract: We introduce a novel method to simultaneously perform variable selection and estimation in the joint frailty model of recurrent and terminal events using the Broken Adaptive Ridge Regression penalty. The BAR penalty can be summarized as an iteratively reweighted squared $L_2$-penalized regression, which approximates the $L_0$-regularization method. Our method allows for the number of covariates to… ▽ More We introduce a novel method to simultaneously perform variable selection and estimation in the joint frailty model of recurrent and terminal events using the Broken Adaptive Ridge Regression penalty. The BAR penalty can be summarized as an iteratively reweighted squared $L_2$-penalized regression, which approximates the $L_0$-regularization method. Our method allows for the number of covariates to diverge with the sample size. Under certain regularity conditions, we prove that the BAR estimator implemented under the model framework is consistent and asymptotically normally distributed, which are known as the oracle properties in the variable selection literature. In our simulation studies, we compare our proposed method to the Minimum Information Criterion (MIC) method. We apply our method on the Medical Information Mart for Intensive Care (MIMIC-III) database, with the aim of investigating which variables affect the risks of repeated ICU admissions and death during ICU stay. △ Less

Submitted 30 August, 2024; originally announced September 2024.

Change Point Detection in Pairwise Comparison Data with Covariates

Authors: Yi Han, Thomas C. M. Lee

Abstract: This paper introduces the novel piecewise stationary covariate-assisted ranking estimation (PS-CARE) model for analyzing time-evolving pairwise comparison data, enhancing item ranking accuracy through the integration of covariate information. By partitioning the data into distinct, stationary segments, the PS-CARE model adeptly detects temporal shifts in item rankings, known as change points, whos… ▽ More This paper introduces the novel piecewise stationary covariate-assisted ranking estimation (PS-CARE) model for analyzing time-evolving pairwise comparison data, enhancing item ranking accuracy through the integration of covariate information. By partitioning the data into distinct, stationary segments, the PS-CARE model adeptly detects temporal shifts in item rankings, known as change points, whose number and positions are initially unknown. Leveraging the minimum description length (MDL) principle, this paper establishes a statistically consistent model selection criterion to estimate these unknowns. The practical optimization of this MDL criterion is done with the pruned exact linear time (PELT) algorithm. Empirical evaluations reveal the method's promising performance in accurately locating change points across various simulated scenarios. An application to an NBA dataset yielded meaningful insights that aligned with significant historical events, highlighting the method's practical utility and the MDL criterion's effectiveness in capturing temporal ranking changes. To the best of the authors' knowledge, this research pioneers change point detection in pairwise comparison data with covariate information, representing a significant leap forward in the field of dynamic ranking analysis. △ Less

Submitted 24 August, 2024; originally announced August 2024.

GenTwoArmsTrialSize: An R Statistical Software Package to estimate Generalized Two Arms Randomized Clinical Trial Sample Size

Authors: Mohsen Soltanifar, Chel Hee Lee, Amin Shirazi, Martha Behnke, Ilfra Raymond-Loher, Getachew A. Dagne

Abstract: The precise calculation of sample sizes is a crucial aspect in the design of clinical trials particularly for pharmaceutical statisticians. While various R statistical software packages have been developed by researchers to estimate required sample sizes under different assumptions, there has been a notable absence of a standalone R statistical software package that allows researchers to comprehen… ▽ More The precise calculation of sample sizes is a crucial aspect in the design of clinical trials particularly for pharmaceutical statisticians. While various R statistical software packages have been developed by researchers to estimate required sample sizes under different assumptions, there has been a notable absence of a standalone R statistical software package that allows researchers to comprehensively estimate sample sizes under generalized scenarios. This paper introduces the R statistical software package "GenTwoArmsTrialSize" available on the Comprehensive R Archive Network (CRAN), designed for estimating the required sample size in two-arm clinical trials. The package incorporates four endpoint types, two trial treatment designs, four types of hypothesis tests, as well as considerations for noncompliance and loss of follow-up, providing researchers with the capability to estimate sample sizes across 24 scenarios. To facilitate understanding of the estimation process and illuminate the impact of noncompliance and loss of follow-up on the size and variability of estimations, the paper includes four hypothetical examples and one applied example. The discussion encompasses the package's limitations and outlines directions for future extensions and improvements. △ Less

Submitted 15 July, 2024; originally announced July 2024.

Comments: 33 pages, 2 figures, 2 tables

MSC Class: 62-04; 62-08; 62K05

Uniform Transformation: Refining Latent Representation in Variational Autoencoders

Authors: Ye Shi, C. S. George Lee

Abstract: Irregular distribution in latent space causes posterior collapse, misalignment between posterior and prior, and ill-sampling problem in Variational Autoencoders (VAEs). In this paper, we introduce a novel adaptable three-stage Uniform Transformation (UT) module -- Gaussian Kernel Density Estimation (G-KDE) clustering, non-parametric Gaussian Mixture (GM) Modeling, and Probability Integral Transfor… ▽ More Irregular distribution in latent space causes posterior collapse, misalignment between posterior and prior, and ill-sampling problem in Variational Autoencoders (VAEs). In this paper, we introduce a novel adaptable three-stage Uniform Transformation (UT) module -- Gaussian Kernel Density Estimation (G-KDE) clustering, non-parametric Gaussian Mixture (GM) Modeling, and Probability Integral Transform (PIT) -- to address irregular latent distributions. By reconfiguring irregular distributions into a uniform distribution in the latent space, our approach significantly enhances the disentanglement and interpretability of latent representations, overcoming the limitation of traditional VAE models in capturing complex data structures. Empirical evaluations demonstrated the efficacy of our proposed UT module in improving disentanglement metrics across benchmark datasets -- dSprites and MNIST. Our findings suggest a promising direction for advancing representation learning techniques, with implication for future research in extending this framework to more sophisticated datasets and downstream tasks. △ Less

Submitted 2 July, 2024; originally announced July 2024.

Comments: Accepted by 2024 IEEE 20th International Conference on Automation Science and Engineering

Adjusting for Selection Bias Due to Missing Eligibility Criteria in Emulated Target Trials

Authors: Luke Benz, Rajarshi Mukherjee, Rui Wang, David Arterburn, Catherine Lee, Heidi Fischer, Susan Shortreed, Sebastien Haneuse

Abstract: Target trial emulation (TTE) is a popular framework for observational studies based on electronic health records (EHR). A key component of this framework is determining the patient population eligible for inclusion in both a target trial of interest and its observational emulation. Missingness in variables that define eligibility criteria, however, presents a major challenge towards determining th… ▽ More Target trial emulation (TTE) is a popular framework for observational studies based on electronic health records (EHR). A key component of this framework is determining the patient population eligible for inclusion in both a target trial of interest and its observational emulation. Missingness in variables that define eligibility criteria, however, presents a major challenge towards determining the eligible population when emulating a target trial with an observational study. In practice, patients with incomplete data are almost always excluded from analysis despite the possibility of selection bias, which can arise when subjects with observed eligibility data are fundamentally different than excluded subjects. Despite this, to the best of our knowledge, very little work has been done to mitigate this concern. In this paper, we propose a novel conceptual framework to address selection bias in TTE studies, tailored towards time-to-event endpoints, and describe estimation and inferential procedures via inverse probability weighting (IPW). Under an EHR-based simulation infrastructure, developed to reflect the complexity of EHR data, we characterize common settings under which missing eligibility data poses the threat of selection bias and investigate the ability of the proposed methods to address it. Finally, using EHR databases from Kaiser Permanente, we demonstrate the use of our method to evaluate the effect of bariatric surgery on microvascular outcomes among a cohort of severely obese patients with Type II diabetes mellitus (T2DM). △ Less

Submitted 4 October, 2024; v1 submitted 24 June, 2024; originally announced June 2024.

Biomarker based Cancer Classification using an Ensemble with Pre-trained Models

Authors: Chongmin Lee, Jihie Kim

Abstract: Certain cancer types, namely pancreatic cancer is difficult to detect at an early stage; sparking the importance of discovering the causal relationship between biomarkers and cancer to identify cancer efficiently. By allowing for the detection and monitoring of specific biomarkers through a non-invasive method, liquid biopsies enhance the precision and efficacy of medical interventions, advocating… ▽ More Certain cancer types, namely pancreatic cancer is difficult to detect at an early stage; sparking the importance of discovering the causal relationship between biomarkers and cancer to identify cancer efficiently. By allowing for the detection and monitoring of specific biomarkers through a non-invasive method, liquid biopsies enhance the precision and efficacy of medical interventions, advocating the move towards personalized healthcare. Several machine learning algorithms such as Random Forest, SVM are utilized for classification, yet causing inefficiency due to the need for conducting hyperparameter tuning. We leverage a meta-trained Hyperfast model for classifying cancer, accomplishing the highest AUC of 0.9929 and simultaneously achieving robustness especially on highly imbalanced datasets compared to other ML algorithms in several binary classification tasks (e.g. breast invasive carcinoma; BRCA vs. non-BRCA). We also propose a novel ensemble model combining pre-trained Hyperfast model, XGBoost, and LightGBM for multi-class classification tasks, achieving an incremental increase in accuracy (0.9464) while merely using 500 PCA features; distinguishable from previous studies where they used more than 2,000 features for similar results. △ Less

Submitted 14 June, 2024; originally announced June 2024.

Comments: Accepted to the AIAA Workshop at IJCAI 2024

Low-rank Matrix Bandits with Heavy-tailed Rewards

Authors: Yue Kang, Cho-Jui Hsieh, Thomas C. M. Lee

Abstract: In stochastic low-rank matrix bandit, the expected reward of an arm is equal to the inner product between its feature matrix and some unknown $d_1$ by $d_2$ low-rank parameter matrix $Θ^*$ with rank $r \ll d_1\wedge d_2$. While all prior studies assume the payoffs are mixed with sub-Gaussian noises, in this work we loosen this strict assumption and consider the new problem of \underline{low}-rank… ▽ More In stochastic low-rank matrix bandit, the expected reward of an arm is equal to the inner product between its feature matrix and some unknown $d_1$ by $d_2$ low-rank parameter matrix $Θ^*$ with rank $r \ll d_1\wedge d_2$. While all prior studies assume the payoffs are mixed with sub-Gaussian noises, in this work we loosen this strict assumption and consider the new problem of \underline{low}-rank matrix bandit with \underline{h}eavy-\underline{t}ailed \underline{r}ewards (LowHTR), where the rewards only have finite $(1+δ)$ moment for some $δ\in (0,1]$. By utilizing the truncation on observed payoffs and the dynamic exploration, we propose a novel algorithm called LOTUS attaining the regret bound of order $\tilde O(d^\frac{3}{2}r^\frac{1}{2}T^\frac{1}{1+δ}/\tilde{D}_{rr})$ without knowing $T$, which matches the state-of-the-art regret bound under sub-Gaussian noises~\citep{lu2021low,kang2022efficient} with $δ= 1$. Moreover, we establish a lower bound of the order $Ω(d^\fracδ{1+δ} r^\fracδ{1+δ} T^\frac{1}{1+δ}) = Ω(T^\frac{1}{1+δ})$ for LowHTR, which indicates our LOTUS is nearly optimal in the order of $T$. In addition, we improve LOTUS so that it does not require knowledge of the rank $r$ with $\tilde O(dr^\frac{3}{2}T^\frac{1+δ}{1+2δ})$ regret bound, and it is efficient under the high-dimensional scenario. We also conduct simulations to demonstrate the practical superiority of our algorithm. △ Less

Submitted 26 April, 2024; originally announced April 2024.

Comments: The 40th Conference on Uncertainty in Artificial Intelligence (UAI 2024)

Double Robust Variance Estimation with Parametric Working Models

Authors: Bonnie E. Shook-Sa, Paul N. Zivich, Chanhwa Lee, Keyi Xue, Rachael K. Ross, Jessie K. Edwards, Jeffrey S. A. Stringer, Stephen R. Cole

Abstract: Doubly robust estimators have gained popularity in the field of causal inference due to their ability to provide consistent point estimates when either an outcome or exposure model is correctly specified. However, for nonrandomized exposures the influence function based variance estimator frequently used with doubly robust estimators of the average causal effect is only consistent when both workin… ▽ More Doubly robust estimators have gained popularity in the field of causal inference due to their ability to provide consistent point estimates when either an outcome or exposure model is correctly specified. However, for nonrandomized exposures the influence function based variance estimator frequently used with doubly robust estimators of the average causal effect is only consistent when both working models (i.e., outcome and exposure models) are correctly specified. Here, the empirical sandwich variance estimator and the nonparametric bootstrap are demonstrated to be doubly robust variance estimators. That is, they are expected to provide valid estimates of the variance leading to nominal confidence interval coverage when only one working model is correctly specified. Simulation studies illustrate the properties of the influence function based, empirical sandwich, and nonparametric bootstrap variance estimators in the setting where parametric working models are assumed. Estimators are applied to data from the Improving Pregnancy Outcomes with Progesterone (IPOP) study to estimate the effect of maternal anemia on birth weight among women with HIV. △ Less

Submitted 4 November, 2024; v1 submitted 24 April, 2024; originally announced April 2024.

Comments: 32 pages, 14 figures, 13 tables

AutoGFI: Streamlined Generalized Fiducial Inference for Modern Inference Problems in Models with Additive Errors

Authors: Wei Du, Jan Hannig, Thomas C. M. Lee, Yi Su, Chunzhe Zhang

Abstract: The concept of fiducial inference was introduced by R. A. Fisher in the 1930s to address the perceived limitations of Bayesian inference, particularly the need for subjective prior distributions in cases with limited prior information. However, Fisher's fiducial approach lost favor due to complications, especially in multi-parameter problems. With renewed interest in fiducial inference in the 2000… ▽ More The concept of fiducial inference was introduced by R. A. Fisher in the 1930s to address the perceived limitations of Bayesian inference, particularly the need for subjective prior distributions in cases with limited prior information. However, Fisher's fiducial approach lost favor due to complications, especially in multi-parameter problems. With renewed interest in fiducial inference in the 2000s, generalized fiducial inference (GFI) emerged as a promising extension of Fisher's ideas, offering new solutions for complex inference challenges. Despite its potential, GFI's adoption has been hindered by demanding mathematical derivations and complex implementation requirements, such as Markov Chain Monte Carlo (MCMC) algorithms. This paper introduces AutoGFI, a streamlined variant of GFI designed to simplify its application across various inference problems with additive noise. AutoGFI's accessibility lies in its simplicity-requiring only a fitting routine-making it a feasible option for a wider range of researchers and practitioners. To demonstrate its efficacy, AutoGFI is applied to three challenging problems: tensor regression, matrix completion, and network cohesion regression. These case studies showcase AutoGFI's competitive performance against specialized solutions, highlighting its potential to broaden the application of GFI in practical domains, ultimately enriching the statistical inference toolkit. △ Less

Submitted 24 December, 2024; v1 submitted 11 April, 2024; originally announced April 2024.

Logistic-beta processes for dependent random probabilities with beta marginals

Authors: Changwoo J. Lee, Alessandro Zito, Huiyan Sang, David B. Dunson

Abstract: The beta distribution serves as a canonical tool for modeling probabilities in statistics and machine learning. However, there is limited work on flexible and computationally convenient stochastic process extensions for modeling dependent random probabilities. We propose a novel stochastic process called the logistic-beta process, whose logistic transformation yields a stochastic process with comm… ▽ More The beta distribution serves as a canonical tool for modeling probabilities in statistics and machine learning. However, there is limited work on flexible and computationally convenient stochastic process extensions for modeling dependent random probabilities. We propose a novel stochastic process called the logistic-beta process, whose logistic transformation yields a stochastic process with common beta marginals. Logistic-beta processes can model dependence on both discrete and continuous domains, such as space or time, and have a flexible dependence structure through correlation kernels. Moreover, its normal variance-mean mixture representation leads to effective posterior inference algorithms. We show how the proposed logistic-beta process can be used to design computationally tractable dependent Bayesian nonparametric models, including dependent Dirichlet processes and extensions. We illustrate the benefits through nonparametric binary regression and conditional density estimation examples, both in simulation studies and in a pregnancy outcome application. △ Less

Submitted 16 March, 2025; v1 submitted 10 February, 2024; originally announced February 2024.

Efficient Frameworks for Generalized Low-Rank Matrix Bandit Problems

Authors: Yue Kang, Cho-Jui Hsieh, Thomas C. M. Lee

Abstract: In the stochastic contextual low-rank matrix bandit problem, the expected reward of an action is given by the inner product between the action's feature matrix and some fixed, but initially unknown $d_1$ by $d_2$ matrix $Θ^*$ with rank $r \ll \{d_1, d_2\}$, and an agent sequentially takes actions based on past experience to maximize the cumulative reward. In this paper, we study the generalized lo… ▽ More In the stochastic contextual low-rank matrix bandit problem, the expected reward of an action is given by the inner product between the action's feature matrix and some fixed, but initially unknown $d_1$ by $d_2$ matrix $Θ^*$ with rank $r \ll \{d_1, d_2\}$, and an agent sequentially takes actions based on past experience to maximize the cumulative reward. In this paper, we study the generalized low-rank matrix bandit problem, which has been recently proposed in \cite{lu2021low} under the Generalized Linear Model (GLM) framework. To overcome the computational infeasibility and theoretical restrain of existing algorithms on this problem, we first propose the G-ESTT framework that modifies the idea from \cite{jun2019bilinear} by using Stein's method on the subspace estimation and then leverage the estimated subspaces via a regularization idea. Furthermore, we remarkably improve the efficiency of G-ESTT by using a novel exclusion idea on the estimated subspace instead, and propose the G-ESTS framework. We also show that G-ESTT can achieve the $\tilde{O}(\sqrt{(d_1+d_2)MrT})$ bound of regret while G-ESTS can achineve the $\tilde{O}(\sqrt{(d_1+d_2)^{3/2}Mr^{3/2}T})$ bound of regret under mild assumption up to logarithm terms, where $M$ is some problem dependent value. Under a reasonable assumption that $M = O((d_1+d_2)^2)$ in our problem setting, the regret of G-ESTT is consistent with the current best regret of $\tilde{O}((d_1+d_2)^{3/2} \sqrt{rT}/D_{rr})$~\citep{lu2021low} ($D_{rr}$ will be defined later). For completeness, we conduct experiments to illustrate that our proposed algorithms, especially G-ESTS, are also computationally tractable and consistently outperform other state-of-the-art (generalized) linear matrix bandit methods based on a suite of simulations. △ Less

Submitted 14 January, 2024; originally announced January 2024.

Comments: Revision of the paper accepted by NeurIPS 2022

A scalable two-stage Bayesian approach accounting for exposure measurement error in environmental epidemiology

Authors: Changwoo J. Lee, Elaine Symanski, Amal Rammah, Dong Hun Kang, Philip K. Hopke, Eun Sug Park

Abstract: Accounting for exposure measurement errors has been recognized as a crucial problem in environmental epidemiology for over two decades. Bayesian hierarchical models offer a coherent probabilistic framework for evaluating associations between environmental exposures and health effects, which take into account exposure measurement errors introduced by uncertainty in the estimated exposure as well as… ▽ More Accounting for exposure measurement errors has been recognized as a crucial problem in environmental epidemiology for over two decades. Bayesian hierarchical models offer a coherent probabilistic framework for evaluating associations between environmental exposures and health effects, which take into account exposure measurement errors introduced by uncertainty in the estimated exposure as well as spatial misalignment between the exposure and health outcome data. While two-stage Bayesian analyses are often regarded as a good alternative to fully Bayesian analyses when joint estimation is not feasible, there has been minimal research on how to properly propagate uncertainty from the first-stage exposure model to the second-stage health model, especially in the case of a large number of participant locations along with spatially correlated exposures. We propose a scalable two-stage Bayesian approach, called a sparse multivariate normal (sparse MVN) prior approach, based on the Vecchia approximation for assessing associations between exposure and health outcomes in environmental epidemiology. We compare its performance with existing approaches through simulation. Our sparse MVN prior approach shows comparable performance with the fully Bayesian approach, which is a gold standard but is impossible to implement in some cases. We investigate the association between source-specific exposures and pollutant (nitrogen dioxide (NO$_2$))-specific exposures and birth outcomes for 2012 in Harris County, Texas, using several approaches, including the newly developed method. △ Less

Submitted 13 January, 2024; v1 submitted 31 December, 2023; originally announced January 2024.

Comments: 34 pages, 8 figures

Clustering Mixtures of Bounded Covariance Distributions Under Optimal Separation

Authors: Ilias Diakonikolas, Daniel M. Kane, Jasper C. H. Lee, Thanasis Pittas

Abstract: We study the clustering problem for mixtures of bounded covariance distributions, under a fine-grained separation assumption. Specifically, given samples from a $k$-component mixture distribution $D = \sum_{i =1}^k w_i P_i$, where each $w_i \ge α$ for some known parameter $α$, and each $P_i$ has unknown covariance $Σ_i \preceq σ^2_i \cdot I_d$ for some unknown $σ_i$, the goal is to cluster the sam… ▽ More We study the clustering problem for mixtures of bounded covariance distributions, under a fine-grained separation assumption. Specifically, given samples from a $k$-component mixture distribution $D = \sum_{i =1}^k w_i P_i$, where each $w_i \ge α$ for some known parameter $α$, and each $P_i$ has unknown covariance $Σ_i \preceq σ^2_i \cdot I_d$ for some unknown $σ_i$, the goal is to cluster the samples assuming a pairwise mean separation in the order of $(σ_i+σ_j)/\sqrtα$ between every pair of components $P_i$ and $P_j$. Our contributions are as follows: For the special case of nearly uniform mixtures, we give the first poly-time algorithm for this clustering task. Prior work either required separation scaling with the maximum cluster standard deviation (i.e. $\max_i σ_i$) [DKK+22b] or required both additional structural assumptions and mean separation scaling as a large degree polynomial in $1/α$ [BKK22]. For general-weight mixtures, we point out that accurate clustering is information-theoretically impossible under our fine-grained mean separation assumptions. We introduce the notion of a clustering refinement -- a list of not-too-small subsets satisfying a similar separation, and which can be merged into a clustering approximating the ground truth -- and show that it is possible to efficiently compute an accurate clustering refinement of the samples. Furthermore, under a variant of the "no large sub-cluster'' condition from in prior work [BKK22], we show that our algorithm outputs an accurate clustering, not just a refinement, even for general-weight mixtures. As a corollary, we obtain efficient clustering algorithms for mixtures of well-conditioned high-dimensional log-concave distributions. Moreover, our algorithm is robust to $Ω(α)$-fraction of adversarial outliers. △ Less

Submitted 18 December, 2023; originally announced December 2023.

Loss-based Objective and Penalizing Priors for Model Selection Problems

Authors: Changwoo J. Lee

Abstract: Many Bayesian model selection problems, such as variable selection or cluster analysis, start by setting prior model probabilities on a structured model space. Based on a chosen loss function between models, model selection is often performed with a Bayes estimator that minimizes the posterior expected loss. The prior model probabilities and the choice of loss both highly affect the model selectio… ▽ More Many Bayesian model selection problems, such as variable selection or cluster analysis, start by setting prior model probabilities on a structured model space. Based on a chosen loss function between models, model selection is often performed with a Bayes estimator that minimizes the posterior expected loss. The prior model probabilities and the choice of loss both highly affect the model selection results, especially for data with small sample sizes, and their proper calibration and careful reflection of no prior model preference are crucial in objective Bayesian analysis. We propose risk equilibrium priors as an objective choice for prior model probabilities that only depend on the model space and the choice of loss. Under the risk equilibrium priors, the Bayes action becomes indifferent before observing data, and the family of the risk equilibrium priors includes existing popular objective priors in Bayesian variable selection problems. We generalize the result to the elicitation of objective priors for Bayesian cluster analysis with Binder's loss. We also propose risk penalization priors, where the Bayes action chooses the simplest model before seeing data. The concept of risk equilibrium and penalization priors allows us to interpret prior properties in light of the effect of loss functions, and also provides new insight into the sensitivity of Bayes estimators under the same prior but different loss. We illustrate the proposed concepts with variable selection simulation studies and cluster analysis on a galaxy dataset. △ Less

Submitted 22 November, 2023; originally announced November 2023.

Comments: 31 pages, 3 figures

Optimality in Mean Estimation: Beyond Worst-Case, Beyond Sub-Gaussian, and Beyond $1+α$ Moments

Authors: Trung Dang, Jasper C. H. Lee, Maoyuan Song, Paul Valiant

Abstract: There is growing interest in improving our algorithmic understanding of fundamental statistical problems such as mean estimation, driven by the goal of understanding the limits of what we can extract from valuable data. The state of the art results for mean estimation in $\mathbb{R}$ are 1) the optimal sub-Gaussian mean estimator by [LV22], with the tight sub-Gaussian constant for all distribution… ▽ More There is growing interest in improving our algorithmic understanding of fundamental statistical problems such as mean estimation, driven by the goal of understanding the limits of what we can extract from valuable data. The state of the art results for mean estimation in $\mathbb{R}$ are 1) the optimal sub-Gaussian mean estimator by [LV22], with the tight sub-Gaussian constant for all distributions with finite but unknown variance, and 2) the analysis of the median-of-means algorithm by [BCL13] and a lower bound by [DLLO16], characterizing the big-O optimal errors for distributions for which only a $1+α$ moment exists for $α\in (0,1)$. Both results, however, are optimal only in the worst case. We initiate the fine-grained study of the mean estimation problem: Can algorithms leverage useful features of the input distribution to beat the sub-Gaussian rate, without explicit knowledge of such features? We resolve this question with an unexpectedly nuanced answer: "Yes in limited regimes, but in general no". For any distribution $p$ with a finite mean, we construct a distribution $q$ whose mean is well-separated from $p$'s, yet $p$ and $q$ are not distinguishable with high probability, and $q$ further preserves $p$'s moments up to constants. The main consequence is that no reasonable estimator can asymptotically achieve better than the sub-Gaussian error rate for any distribution, matching the worst-case result of [LV22]. More generally, we introduce a new definitional framework to analyze the fine-grained optimality of algorithms, which we call "neighborhood optimality", interpolating between the unattainably strong "instance optimality" and the trivially weak "admissibility" definitions. Applying the new framework, we show that median-of-means is neighborhood optimal, up to constant factors. It is open to find a neighborhood-optimal estimator without constant factor slackness. △ Less

Submitted 21 November, 2023; originally announced November 2023.

Comments: 27 pages, to appear in NeurIPS 2023. Abstract shortened to fit arXiv limit

Causal Quantile Treatment Effects with missing data by double-sampling

Authors: Shuo Sun, Sebastien Haneuse, Alexander W. Levis, Catherine Lee, David E Arterburn, Heidi Fischer, Susan Shortreed, Rajarshi Mukherjee

Abstract: Causal weighted quantile treatment effects (WQTE) are a useful complement to standard causal contrasts that focus on the mean when interest lies at the tails of the counterfactual distribution. To-date, however, methods for estimation and inference regarding causal WQTEs have assumed complete data on all relevant factors. In most practical settings, however, data will be missing or incomplete data… ▽ More Causal weighted quantile treatment effects (WQTE) are a useful complement to standard causal contrasts that focus on the mean when interest lies at the tails of the counterfactual distribution. To-date, however, methods for estimation and inference regarding causal WQTEs have assumed complete data on all relevant factors. In most practical settings, however, data will be missing or incomplete data, particularly when the data are not collected for research purposes, as is the case for electronic health records and disease registries. Furthermore, such data sources may be particularly susceptible to the outcome data being missing-not-at-random (MNAR). In this paper, we consider the use of double-sampling, through which the otherwise missing data are ascertained on a sub-sample of study units, as a strategy to mitigate bias due to MNAR data in the estimation of causal WQTEs. With the additional data in-hand, we present identifying conditions that do not require assumptions regarding missingness in the original data. We then propose a novel inverse-probability weighted estimator and derive its asymptotic properties, both pointwise at specific quantiles and uniformly across a range of quantiles over some compact subset of (0,1), allowing the propensity score and double-sampling probabilities to be estimated. For practical inference, we develop a bootstrap method that can be used for both pointwise and uniform inference. A simulation study is conducted to examine the finite sample performance of the proposed estimators. The proposed method is illustrated with data from an EHR-based study examining the relative effects of two bariatric surgery procedures on BMI loss at 3 years post-surgery. △ Less

Submitted 14 March, 2025; v1 submitted 13 October, 2023; originally announced October 2023.

Development and validation of an interpretable machine learning-based calculator for predicting 5-year weight trajectories after bariatric surgery: a multinational retrospective cohort SOPHIA study

Authors: Patrick Saux, Pierre Bauvin, Violeta Raverdy, Julien Teigny, Hélène Verkindt, Tomy Soumphonphakdy, Maxence Debert, Anne Jacobs, Daan Jacobs, Valerie Monpellier, Phong Ching Lee, Chin Hong Lim, Johanna C Andersson-Assarsson, Lena Carlsson, Per-Arne Svensson, Florence Galtier, Guelareh Dezfoulian, Mihaela Moldovanu, Severine Andrieux, Julien Couster, Marie Lepage, Erminia Lembo, Ornella Verrastro, Maud Robert, Paulina Salminen , et al. (9 additional authors not shown)

Abstract: Background Weight loss trajectories after bariatric surgery vary widely between individuals, and predicting weight loss before the operation remains challenging. We aimed to develop a model using machine learning to provide individual preoperative prediction of 5-year weight loss trajectories after surgery. Methods In this multinational retrospective observational study we enrolled adult participa… ▽ More Background Weight loss trajectories after bariatric surgery vary widely between individuals, and predicting weight loss before the operation remains challenging. We aimed to develop a model using machine learning to provide individual preoperative prediction of 5-year weight loss trajectories after surgery. Methods In this multinational retrospective observational study we enrolled adult participants (aged $\ge$18 years) from ten prospective cohorts (including ABOS [NCT01129297], BAREVAL [NCT02310178], the Swedish Obese Subjects study, and a large cohort from the Dutch Obesity Clinic [Nederlandse Obesitas Kliniek]) and two randomised trials (SleevePass [NCT00793143] and SM-BOSS [NCT00356213]) in Europe, the Americas, and Asia, with a 5 year followup after Roux-en-Y gastric bypass, sleeve gastrectomy, or gastric band. Patients with a previous history of bariatric surgery or large delays between scheduled and actual visits were excluded. The training cohort comprised patients from two centres in France (ABOS and BAREVAL). The primary outcome was BMI at 5 years. A model was developed using least absolute shrinkage and selection operator to select variables and the classification and regression trees algorithm to build interpretable regression trees. The performances of the model were assessed through the median absolute deviation (MAD) and root mean squared error (RMSE) of BMI. Findings10 231 patients from 12 centres in ten countries were included in the analysis, corresponding to 30 602 patient-years. Among participants in all 12 cohorts, 7701 (75$\bullet$3%) were female, 2530 (24$\bullet$7%) were male. Among 434 baseline attributes available in the training cohort, seven variables were selected: height, weight, intervention type, age, diabetes status, diabetes duration, and smoking status. At 5 years, across external testing cohorts the overall mean MAD BMI was 2$\bullet$8 kg/m${}^2$ (95% CI 2$\bullet$6-3$\bullet$0) and mean RMSE BMI was 4$\bullet$7 kg/m${}^2$ (4$\bullet$4-5$\bullet$0), and the mean difference between predicted and observed BMI was-0$\bullet$3 kg/m${}^2$ (SD 4$\bullet$7). This model is incorporated in an easy to use and interpretable web-based prediction tool to help inform clinical decision before surgery. InterpretationWe developed a machine learning-based model, which is internationally validated, for predicting individual 5-year weight loss trajectories after three common bariatric interventions. △ Less

Submitted 31 August, 2023; originally announced August 2023.

Comments: The Lancet Digital Health, 2023

Finite-Sample Symmetric Mean Estimation with Fisher Information Rate

Authors: Shivam Gupta, Jasper C. H. Lee, Eric Price

Abstract: The mean of an unknown variance-$σ^2$ distribution $f$ can be estimated from $n$ samples with variance $\frac{σ^2}{n}$ and nearly corresponding subgaussian rate. When $f$ is known up to translation, this can be improved asymptotically to $\frac{1}{n\mathcal I}$, where $\mathcal I$ is the Fisher information of the distribution. Such an improvement is not possible for general unknown $f$, but [Stone… ▽ More The mean of an unknown variance-$σ^2$ distribution $f$ can be estimated from $n$ samples with variance $\frac{σ^2}{n}$ and nearly corresponding subgaussian rate. When $f$ is known up to translation, this can be improved asymptotically to $\frac{1}{n\mathcal I}$, where $\mathcal I$ is the Fisher information of the distribution. Such an improvement is not possible for general unknown $f$, but [Stone, 1975] showed that this asymptotic convergence $\textit{is}$ possible if $f$ is $\textit{symmetric}$ about its mean. Stone's bound is asymptotic, however: the $n$ required for convergence depends in an unspecified way on the distribution $f$ and failure probability $δ$. In this paper we give finite-sample guarantees for symmetric mean estimation in terms of Fisher information. For every $f, n, δ$ with $n > \log \frac{1}δ$, we get convergence close to a subgaussian with variance $\frac{1}{n \mathcal I_r}$, where $\mathcal I_r$ is the $r$-$\textit{smoothed}$ Fisher information with smoothing radius $r$ that decays polynomially in $n$. Such a bound essentially matches the finite-sample guarantees in the known-$f$ setting. △ Less

Submitted 28 June, 2023; originally announced June 2023.

Comments: COLT 2023

Robust Lipschitz Bandits to Adversarial Corruptions

Authors: Yue Kang, Cho-Jui Hsieh, Thomas C. M. Lee

Abstract: Lipschitz bandit is a variant of stochastic bandits that deals with a continuous arm set defined on a metric space, where the reward function is subject to a Lipschitz constraint. In this paper, we introduce a new problem of Lipschitz bandits in the presence of adversarial corruptions where an adaptive adversary corrupts the stochastic rewards up to a total budget $C$. The budget is measured by th… ▽ More Lipschitz bandit is a variant of stochastic bandits that deals with a continuous arm set defined on a metric space, where the reward function is subject to a Lipschitz constraint. In this paper, we introduce a new problem of Lipschitz bandits in the presence of adversarial corruptions where an adaptive adversary corrupts the stochastic rewards up to a total budget $C$. The budget is measured by the sum of corruption levels across the time horizon $T$. We consider both weak and strong adversaries, where the weak adversary is unaware of the current action before the attack, while the strong one can observe it. Our work presents the first line of robust Lipschitz bandit algorithms that can achieve sub-linear regret under both types of adversary, even when the total budget of corruption $C$ is unrevealed to the agent. We provide a lower bound under each type of adversary, and show that our algorithm is optimal under the strong case. Finally, we conduct experiments to illustrate the effectiveness of our algorithms against two classic kinds of attacks. △ Less

Submitted 8 October, 2023; v1 submitted 29 May, 2023; originally announced May 2023.

Comments: Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023)

Training Energy-Based Normalizing Flow with Score-Matching Objectives

Authors: Chen-Hao Chao, Wei-Fang Sun, Yen-Chang Hsu, Zsolt Kira, Chun-Yi Lee

Abstract: In this paper, we establish a connection between the parameterization of flow-based and energy-based generative models, and present a new flow-based modeling approach called energy-based normalizing flow (EBFlow). We demonstrate that by optimizing EBFlow with score-matching objectives, the computation of Jacobian determinants for linear transformations can be entirely bypassed. This feature enable… ▽ More In this paper, we establish a connection between the parameterization of flow-based and energy-based generative models, and present a new flow-based modeling approach called energy-based normalizing flow (EBFlow). We demonstrate that by optimizing EBFlow with score-matching objectives, the computation of Jacobian determinants for linear transformations can be entirely bypassed. This feature enables the use of arbitrary linear layers in the construction of flow-based models without increasing the computational time complexity of each training iteration from $O(D^2L)$ to $O(D^3L)$ for an $L$-layered model that accepts $D$-dimensional inputs. This makes the training of EBFlow more efficient than the commonly-adopted maximum likelihood training method. In addition to the reduction in runtime, we enhance the training stability and empirical performance of EBFlow through a number of techniques developed based on our analysis of the score-matching methods. The experimental results demonstrate that our approach achieves a significant speedup compared to maximum likelihood estimation while outperforming prior methods with a noticeable margin in terms of negative log-likelihood (NLL). △ Less

Submitted 28 October, 2023; v1 submitted 24 May, 2023; originally announced May 2023.

Comments: Published at NeurIPS 2023. Code: https://github.com/chen-hao-chao/ebflow

Probabilistic forecast of nonlinear dynamical systems with uncertainty quantification

Authors: Mengyang Gu, Yizi Lin, Victor Chang Lee, Diana Qiu

Abstract: Data-driven modeling is useful for reconstructing nonlinear dynamical systems when the underlying process is unknown or too expensive to compute. Having reliable uncertainty assessment of the forecast enables tools to be deployed to predict new scenarios unobserved before. In this work, we first extend parallel partial Gaussian processes for predicting the vector-valued transition function that li… ▽ More Data-driven modeling is useful for reconstructing nonlinear dynamical systems when the underlying process is unknown or too expensive to compute. Having reliable uncertainty assessment of the forecast enables tools to be deployed to predict new scenarios unobserved before. In this work, we first extend parallel partial Gaussian processes for predicting the vector-valued transition function that links the observations between the current and next time points, and quantify the uncertainty of predictions by posterior sampling. Second, we show the equivalence between the dynamic mode decomposition and the maximum likelihood estimator of the linear mapping matrix in the linear state space model. The connection provides a {probabilistic generative} model of dynamic mode decomposition and thus, uncertainty of predictions can be obtained. Furthermore, we draw close connections between different data-driven models for approximating nonlinear dynamics, through a unified view of generative models. We study two numerical examples, where the inputs of the dynamics are assumed to be known in the first example and the inputs are unknown in the second example. The examples indicate that uncertainty of forecast can be properly quantified, whereas model or input misspecification can degrade the accuracy of uncertainty quantification. △ Less

Submitted 30 October, 2023; v1 submitted 15 May, 2023; originally announced May 2023.

Journal ref: Physica D: Nonlinear Phenomena, 133938 (2023)

A Spectral Algorithm for List-Decodable Covariance Estimation in Relative Frobenius Norm

Authors: Ilias Diakonikolas, Daniel M. Kane, Jasper C. H. Lee, Ankit Pensia, Thanasis Pittas

Abstract: We study the problem of list-decodable Gaussian covariance estimation. Given a multiset $T$ of $n$ points in $\mathbb R^d$ such that an unknown $α<1/2$ fraction of points in $T$ are i.i.d. samples from an unknown Gaussian $\mathcal{N}(μ, Σ)$, the goal is to output a list of $O(1/α)$ hypotheses at least one of which is close to $Σ$ in relative Frobenius norm. Our main result is a… ▽ More We study the problem of list-decodable Gaussian covariance estimation. Given a multiset $T$ of $n$ points in $\mathbb R^d$ such that an unknown $α<1/2$ fraction of points in $T$ are i.i.d. samples from an unknown Gaussian $\mathcal{N}(μ, Σ)$, the goal is to output a list of $O(1/α)$ hypotheses at least one of which is close to $Σ$ in relative Frobenius norm. Our main result is a $\mathrm{poly}(d,1/α)$ sample and time algorithm for this task that guarantees relative Frobenius norm error of $\mathrm{poly}(1/α)$. Importantly, our algorithm relies purely on spectral techniques. As a corollary, we obtain an efficient spectral algorithm for robust partial clustering of Gaussian mixture models (GMMs) -- a key ingredient in the recent work of [BDJ+22] on robustly learning arbitrary GMMs. Combined with the other components of [BDJ+22], our new method yields the first Sum-of-Squares-free algorithm for robustly learning GMMs. At the technical level, we develop a novel multi-filtering method for list-decodable covariance estimation that may be useful in other settings. △ Less

Submitted 1 May, 2023; originally announced May 2023.

Statistical and computational rates in high rank tensor estimation

Authors: Chanwoo Lee, Miaoyan Wang

Abstract: Higher-order tensor datasets arise commonly in recommendation systems, neuroimaging, and social networks. Here we develop probable methods for estimating a possibly high rank signal tensor from noisy observations. We consider a generative latent variable tensor model that incorporates both high rank and low rank models, including but not limited to, simple hypergraphon models, single index models,… ▽ More Higher-order tensor datasets arise commonly in recommendation systems, neuroimaging, and social networks. Here we develop probable methods for estimating a possibly high rank signal tensor from noisy observations. We consider a generative latent variable tensor model that incorporates both high rank and low rank models, including but not limited to, simple hypergraphon models, single index models, low-rank CP models, and low-rank Tucker models. Comprehensive results are developed on both the statistical and computational limits for the signal tensor estimation. We find that high-dimensional latent variable tensors are of log-rank; the fact explains the pervasiveness of low-rank tensors in applications. Furthermore, we propose a polynomial-time spectral algorithm that achieves the computationally optimal rate. We show that the statistical-computational gap emerges only for latent variable tensors of order 3 or higher. Numerical experiments and two real data applications are presented to demonstrate the practical merits of our methods. △ Less

Submitted 8 April, 2023; originally announced April 2023.

Comments: 38 pages, 8 figures

Sufficient dimension reduction for feature matrices

Authors: Chanwoo Lee

Abstract: We address the problem of sufficient dimension reduction for feature matrices, which arises often in sensor network localization, brain neuroimaging, and electroencephalography analysis. In general, feature matrices have both row- and column-wise interpretations and contain structural information that can be lost with naive vectorization approaches. To address this, we propose a method called prin… ▽ More We address the problem of sufficient dimension reduction for feature matrices, which arises often in sensor network localization, brain neuroimaging, and electroencephalography analysis. In general, feature matrices have both row- and column-wise interpretations and contain structural information that can be lost with naive vectorization approaches. To address this, we propose a method called principal support matrix machine (PSMM) for the matrix sufficient dimension reduction. The PSMM converts the sufficient dimension reduction problem into a series of classification problems by dividing the response variables into slices. It effectively utilizes the matrix structure by finding hyperplanes with rank-1 normal matrix that optimally separate the sliced responses. Additionally, we extend our approach to the higher-order tensor case. Our numerical analysis demonstrates that the PSMM outperforms existing methods and has strong interpretability in real data applications. △ Less

Submitted 7 March, 2023; originally announced March 2023.

Comments: 30 pages, 3 figures

Online Continuous Hyperparameter Optimization for Generalized Linear Contextual Bandits

Authors: Yue Kang, Cho-Jui Hsieh, Thomas C. M. Lee

Abstract: In stochastic contextual bandits, an agent sequentially makes actions from a time-dependent action set based on past experience to minimize the cumulative regret. Like many other machine learning algorithms, the performance of bandits heavily depends on the values of hyperparameters, and theoretically derived parameter values may lead to unsatisfactory results in practice. Moreover, it is infeasib… ▽ More In stochastic contextual bandits, an agent sequentially makes actions from a time-dependent action set based on past experience to minimize the cumulative regret. Like many other machine learning algorithms, the performance of bandits heavily depends on the values of hyperparameters, and theoretically derived parameter values may lead to unsatisfactory results in practice. Moreover, it is infeasible to use offline tuning methods like cross-validation to choose hyperparameters under the bandit environment, as the decisions should be made in real-time. To address this challenge, we propose the first online continuous hyperparameter tuning framework for contextual bandits to learn the optimal parameter configuration in practice within a search space on the fly. Specifically, we use a double-layer bandit framework named CDT (Continuous Dynamic Tuning) and formulate the hyperparameter optimization as a non-stationary continuum-armed bandit, where each arm represents a combination of hyperparameters, and the corresponding reward is the algorithmic result. For the top layer, we propose the Zooming TS algorithm that utilizes Thompson Sampling (TS) for exploration and a restart technique to get around the \textit{switching} environment. The proposed CDT framework can be easily utilized to tune contextual bandit algorithms without any pre-specified candidate set for multiple hyperparameters. We further show that it could achieve a sublinear regret in theory and performs consistently better than all existing methods on both synthetic and real datasets. △ Less

Submitted 8 April, 2024; v1 submitted 18 February, 2023; originally announced February 2023.

Comments: Published in Transactions on Machine Learning Research (TMLR)

High-dimensional Location Estimation via Norm Concentration for Subgamma Vectors

Authors: Shivam Gupta, Jasper C. H. Lee, Eric Price

Abstract: In location estimation, we are given $n$ samples from a known distribution $f$ shifted by an unknown translation $λ$, and want to estimate $λ$ as precisely as possible. Asymptotically, the maximum likelihood estimate achieves the Cramér-Rao bound of error $\mathcal N(0, \frac{1}{n\mathcal I})$, where $\mathcal I$ is the Fisher information of $f$. However, the $n$ required for convergence depends o… ▽ More In location estimation, we are given $n$ samples from a known distribution $f$ shifted by an unknown translation $λ$, and want to estimate $λ$ as precisely as possible. Asymptotically, the maximum likelihood estimate achieves the Cramér-Rao bound of error $\mathcal N(0, \frac{1}{n\mathcal I})$, where $\mathcal I$ is the Fisher information of $f$. However, the $n$ required for convergence depends on $f$, and may be arbitrarily large. We build on the theory using \emph{smoothed} estimators to bound the error for finite $n$ in terms of $\mathcal I_r$, the Fisher information of the $r$-smoothed distribution. As $n \to \infty$, $r \to 0$ at an explicit rate and this converges to the Cramér-Rao bound. We (1) improve the prior work for 1-dimensional $f$ to converge for constant failure probability in addition to high probability, and (2) extend the theory to high-dimensional distributions. In the process, we prove a new bound on the norm of a high-dimensional random variable whose 1-dimensional projections are subgamma, which may be of independent interest. △ Less

Submitted 5 February, 2023; originally announced February 2023.

A Bayesian analysis of current duration data with reporting issues: an application to estimating the distribution of time-between-sex from time-since-last-sex data as collected in cross-sectional surveys in low- and middle-income countries

Authors: Chi Hyun Lee, Herbert Susmann, Leontine Alkema

Abstract: Aggregate measures of family planning are used to monitor demand for and usage of contraceptive methods in populations globally, for example as part of the FP2030 initiative. Family planning measures for low- and middle-income countries are typically based on data collected through cross-sectional household surveys. Recently proposed measures account for sexual activity through assessment of the d… ▽ More Aggregate measures of family planning are used to monitor demand for and usage of contraceptive methods in populations globally, for example as part of the FP2030 initiative. Family planning measures for low- and middle-income countries are typically based on data collected through cross-sectional household surveys. Recently proposed measures account for sexual activity through assessment of the distribution of time-between-sex (TBS) in the population of interest. In this paper, we propose a statistical approach to estimate the distribution of TBS using data typically available in low- and middle-income countries, while addressing two major challenges. The first challenge is that timing of sex information is typically limited to women's time-since-last-sex (TSLS) data collected in the cross-sectional survey. In our proposed approach, we adopt the current duration method to estimate the distribution of TBS using the available TSLS data, from which the frequency of sex at the population level can be derived. Furthermore, the observed TSLS data are subject to reporting issues because they can be reported in different units and may be rounded off. To apply the current duration approach and account for these data reporting issues, we develop a flexible Bayesian model, and provide a detailed technical description of the proposed modeling approach. △ Less

Submitted 2 February, 2023; originally announced February 2023.

A Bayesian Nonparametric Stochastic Block Model for Directed Acyclic Graphs

Authors: Clement Lee, Marco Battiston

Abstract: Random graphs have been widely used in statistics, for example in network and social interaction analysis. In some applications, data may contain an inherent hierarchical ordering among its vertices, which prevents any directed edge between pairs of vertices that do not respect this order. For example, in bibliometrics, older papers cannot cite newer ones. In such situations, the resulting graph f… ▽ More Random graphs have been widely used in statistics, for example in network and social interaction analysis. In some applications, data may contain an inherent hierarchical ordering among its vertices, which prevents any directed edge between pairs of vertices that do not respect this order. For example, in bibliometrics, older papers cannot cite newer ones. In such situations, the resulting graph forms a Directed Acyclic Graph. In this article, we propose an extension of the popular Stochastic Block Model (SBM) to account for the presence of a latent hierarchical ordering in the data. The proposed approach includes a topological ordering in the likelihood of the model, which allows a directed edge to have positive probability only if the corresponding pair of vertices respect the order. This latent ordering is treated as an unknown parameter and endowed with a prior distribution. We describe how to formalize the model and perform posterior inference for a Bayesian nonparametric version of the SBM in which both the hierarchical ordering and the number of latent blocks are learnt from the data. Finally, an illustration with a real-world dataset from bibliometrics is presented. Additional supplementary materials are available online. △ Less

Submitted 10 December, 2024; v1 submitted 18 January, 2023; originally announced January 2023.

Comments: 27 pages, 10 figures, 3 tables

Neural Spline Search for Quantile Probabilistic Modeling

Authors: Ruoxi Sun, Chun-Liang Li, Sercan O. Arik, Michael W. Dusenberry, Chen-Yu Lee, Tomas Pfister

Abstract: Accurate estimation of output quantiles is crucial in many use cases, where it is desired to model the range of possibility. Modeling target distribution at arbitrary quantile levels and at arbitrary input attribute levels are important to offer a comprehensive picture of the data, and requires the quantile function to be expressive enough. The quantile function describing the target distribution… ▽ More Accurate estimation of output quantiles is crucial in many use cases, where it is desired to model the range of possibility. Modeling target distribution at arbitrary quantile levels and at arbitrary input attribute levels are important to offer a comprehensive picture of the data, and requires the quantile function to be expressive enough. The quantile function describing the target distribution using quantile levels is critical for quantile regression. Although various parametric forms for the distributions (that the quantile function specifies) can be adopted, an everlasting problem is selecting the most appropriate one that can properly approximate the data distributions. In this paper, we propose a non-parametric and data-driven approach, Neural Spline Search (NSS), to represent the observed data distribution without parametric assumptions. NSS is flexible and expressive for modeling data distributions by transforming the inputs with a series of monotonic spline regressions guided by symbolic operators. We demonstrate that NSS outperforms previous methods on synthetic, real-world regression and time-series forecasting tasks. △ Less

Submitted 12 January, 2023; originally announced January 2023.

Efficient Nonparametric Estimation of Stochastic Policy Effects with Clustered Interference

Authors: Chanhwa Lee, Donglin Zeng, Michael G. Hudgens

Abstract: Interference occurs when a unit's treatment (or exposure) affects another unit's outcome. In some settings, units may be grouped into clusters such that it is reasonable to assume that interference, if present, only occurs between individuals in the same cluster, i.e., there is clustered interference. Various causal estimands have been proposed to quantify treatment effects under clustered interfe… ▽ More Interference occurs when a unit's treatment (or exposure) affects another unit's outcome. In some settings, units may be grouped into clusters such that it is reasonable to assume that interference, if present, only occurs between individuals in the same cluster, i.e., there is clustered interference. Various causal estimands have been proposed to quantify treatment effects under clustered interference from observational data, but these estimands either entail treatment policies lacking real-world relevance or are based on parametric propensity score models. Here, we propose new causal estimands based on modification of the propensity score distribution which may be more relevant in many contexts and are not based on parametric models. Nonparametric sample splitting estimators of the new estimands are constructed, which allow for flexible data-adaptive estimation of nuisance functions and are consistent, asymptotically normal, and efficient, converging at the usual parametric rate. Simulations show the finite sample performance of the proposed estimators. The proposed methods are applied to evaluate the effect of water, sanitation, and hygiene facilities on diarrhea among children in Senegal. △ Less

Submitted 23 August, 2023; v1 submitted 21 December, 2022; originally announced December 2022.

Outlier-Robust Sparse Mean Estimation for Heavy-Tailed Distributions

Authors: Ilias Diakonikolas, Daniel M. Kane, Jasper C. H. Lee, Ankit Pensia

Abstract: We study the fundamental task of outlier-robust mean estimation for heavy-tailed distributions in the presence of sparsity. Specifically, given a small number of corrupted samples from a high-dimensional heavy-tailed distribution whose mean $μ$ is guaranteed to be sparse, the goal is to efficiently compute a hypothesis that accurately approximates $μ$ with high probability. Prior work had obtained… ▽ More We study the fundamental task of outlier-robust mean estimation for heavy-tailed distributions in the presence of sparsity. Specifically, given a small number of corrupted samples from a high-dimensional heavy-tailed distribution whose mean $μ$ is guaranteed to be sparse, the goal is to efficiently compute a hypothesis that accurately approximates $μ$ with high probability. Prior work had obtained efficient algorithms for robust sparse mean estimation of light-tailed distributions. In this work, we give the first sample-efficient and polynomial-time robust sparse mean estimator for heavy-tailed distributions under mild moment assumptions. Our algorithm achieves the optimal asymptotic error using a number of samples scaling logarithmically with the ambient dimension. Importantly, the sample complexity of our method is optimal as a function of the failure probability $τ$, having an additive $\log(1/τ)$ dependence. Our algorithm leverages the stability-based approach from the algorithmic robust statistics literature, with crucial (and necessary) adaptations required in our setting. Our analysis may be of independent interest, involving the delicate design of a (non-spectral) decomposition for positive semi-definite matrices satisfying certain sparsity properties. △ Less

Submitted 29 November, 2022; originally announced November 2022.

Comments: To appear in NeurIPS 2022

Spatial distribution and determinants of childhood vaccination refusal in the United States

Authors: Bokgyeong Kang, Sandra Goldlust, Elizabeth C. Lee, John Hughes, Shweta Bansal, Murali Haran

Abstract: Parental refusal and delay of childhood vaccination has increased in recent years in the United States. This phenomenon challenges maintenance of herd immunity and increases the risk of outbreaks of vaccine-preventable diseases. We examine US county-level vaccine refusal for patients under five years of age collected during the period 2012--2015 from an administrative healthcare dataset. We model… ▽ More Parental refusal and delay of childhood vaccination has increased in recent years in the United States. This phenomenon challenges maintenance of herd immunity and increases the risk of outbreaks of vaccine-preventable diseases. We examine US county-level vaccine refusal for patients under five years of age collected during the period 2012--2015 from an administrative healthcare dataset. We model these data with a Bayesian zero-inflated negative binomial regression model to capture social and political processes that are associated with vaccine refusal, as well as factors that affect our measurement of vaccine refusal.Our work highlights fine-scale socio-demographic characteristics associated with vaccine refusal nationally, finds that spatial clustering in refusal can be explained by such factors, and has the potential to aid in the development of targeted public health strategies for optimizing vaccine uptake. △ Less

Submitted 15 March, 2023; v1 submitted 7 November, 2022; originally announced November 2022.

Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimization

Authors: Xingyou Song, Sagi Perel, Chansoo Lee, Greg Kochanski, Daniel Golovin

Abstract: Vizier is the de-facto blackbox and hyperparameter optimization service across Google, having optimized some of Google's largest products and research efforts. To operate at the scale of tuning thousands of users' critical systems, Google Vizier solved key design challenges in providing multiple different features, while remaining fully fault-tolerant. In this paper, we introduce Open Source (OSS)… ▽ More Vizier is the de-facto blackbox and hyperparameter optimization service across Google, having optimized some of Google's largest products and research efforts. To operate at the scale of tuning thousands of users' critical systems, Google Vizier solved key design challenges in providing multiple different features, while remaining fully fault-tolerant. In this paper, we introduce Open Source (OSS) Vizier, a standalone Python-based interface for blackbox optimization and research, based on the Google-internal Vizier infrastructure and framework. OSS Vizier provides an API capable of defining and solving a wide variety of optimization problems, including multi-metric, early stopping, transfer learning, and conditional search. Furthermore, it is designed to be a distributed system that assures reliability, and allows multiple parallel evaluations of the user's objective function. The flexible RPC-based infrastructure allows users to access OSS Vizier from binaries written in any language. OSS Vizier also provides a back-end ("Pythia") API that gives algorithm authors a way to interface new algorithms with the core OSS Vizier system. OSS Vizier is available at https://github.com/google/vizier. △ Less

Submitted 10 January, 2023; v1 submitted 27 July, 2022; originally announced July 2022.

Comments: Published as a conference paper for the systems track at the 1st International Conference on Automated Machine Learning (AutoML-Conf 2022). Code can be found at https://github.com/google/vizier

Pre-training helps Bayesian optimization too

Authors: Zi Wang, George E. Dahl, Kevin Swersky, Chansoo Lee, Zelda Mariet, Zachary Nado, Justin Gilmer, Jasper Snoek, Zoubin Ghahramani

Abstract: Bayesian optimization (BO) has become a popular strategy for global optimization of many expensive real-world functions. Contrary to a common belief that BO is suited to optimizing black-box functions, it actually requires domain knowledge on characteristics of those functions to deploy BO successfully. Such domain knowledge often manifests in Gaussian process priors that specify initial beliefs o… ▽ More Bayesian optimization (BO) has become a popular strategy for global optimization of many expensive real-world functions. Contrary to a common belief that BO is suited to optimizing black-box functions, it actually requires domain knowledge on characteristics of those functions to deploy BO successfully. Such domain knowledge often manifests in Gaussian process priors that specify initial beliefs on functions. However, even with expert knowledge, it is not an easy task to select a prior. This is especially true for hyperparameter tuning problems on complex machine learning models, where landscapes of tuning objectives are often difficult to comprehend. We seek an alternative practice for setting these functional priors. In particular, we consider the scenario where we have data from similar functions that allow us to pre-train a tighter distribution a priori. To verify our approach in realistic model training setups, we collected a large multi-task hyperparameter tuning dataset by training tens of thousands of configurations of near-state-of-the-art models on popular image and text datasets, as well as a protein sequence dataset. Our results show that on average, our method is able to locate good hyperparameters at least 3 times more efficiently than the best competing methods. △ Less

Submitted 7 July, 2022; originally announced July 2022.

Comments: ICML2022 Workshop on Adaptive Experimental Design and Active Learning in the Real World. arXiv admin note: substantial text overlap with arXiv:2109.08215

Rapidly Mixing Multiple-try Metropolis Algorithms for Model Selection Problems

Authors: Hyunwoong Chang, Changwoo J. Lee, Zhao Tang Luo, Huiyan Sang, Quan Zhou

Abstract: The multiple-try Metropolis (MTM) algorithm is an extension of the Metropolis-Hastings (MH) algorithm by selecting the proposed state among multiple trials according to some weight function. Although MTM has gained great popularity owing to its faster empirical convergence and mixing than the standard MH algorithm, its theoretical mixing property is rarely studied in the literature due to its comp… ▽ More The multiple-try Metropolis (MTM) algorithm is an extension of the Metropolis-Hastings (MH) algorithm by selecting the proposed state among multiple trials according to some weight function. Although MTM has gained great popularity owing to its faster empirical convergence and mixing than the standard MH algorithm, its theoretical mixing property is rarely studied in the literature due to its complex proposal scheme. We prove that MTM can achieve a mixing time bound smaller than that of MH by a factor of the number of trials under a general setting applicable to high-dimensional model selection problems with discrete state spaces. Our theoretical results motivate a new class of weight functions called locally balanced weight functions and guide the choice of the number of trials, which leads to improved performance over standard MTM algorithms. We support our theoretical results by extensive simulation studies and real data applications with several Bayesian model selection problems. △ Less

Submitted 14 October, 2022; v1 submitted 1 July, 2022; originally announced July 2022.

Comments: Accepted to Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2022)

Finite-Sample Maximum Likelihood Estimation of Location

Authors: Shivam Gupta, Jasper C. H. Lee, Eric Price, Paul Valiant

Abstract: We consider 1-dimensional location estimation, where we estimate a parameter $λ$ from $n$ samples $λ+ η_i$, with each $η_i$ drawn i.i.d. from a known distribution $f$. For fixed $f$ the maximum-likelihood estimate (MLE) is well-known to be optimal in the limit as $n \to \infty$: it is asymptotically normal with variance matching the Cramér-Rao lower bound of $\frac{1}{n\mathcal{I}}$, where… ▽ More We consider 1-dimensional location estimation, where we estimate a parameter $λ$ from $n$ samples $λ+ η_i$, with each $η_i$ drawn i.i.d. from a known distribution $f$. For fixed $f$ the maximum-likelihood estimate (MLE) is well-known to be optimal in the limit as $n \to \infty$: it is asymptotically normal with variance matching the Cramér-Rao lower bound of $\frac{1}{n\mathcal{I}}$, where $\mathcal{I}$ is the Fisher information of $f$. However, this bound does not hold for finite $n$, or when $f$ varies with $n$. We show for arbitrary $f$ and $n$ that one can recover a similar theory based on the Fisher information of a smoothed version of $f$, where the smoothing radius decays with $n$. △ Less

Submitted 18 July, 2022; v1 submitted 6 June, 2022; originally announced June 2022.

Comments: Corrected an inaccuracy in the description of the experimental setup. Also updated funding acknowledgements

Towards Learning Universal Hyperparameter Optimizers with Transformers

Authors: Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Qiuyi Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Marc'aurelio Ranzato, Sagi Perel, Nando de Freitas

Abstract: Meta-learning hyperparameter optimization (HPO) algorithms from prior experiments is a promising approach to improve optimization efficiency over objective functions from a similar distribution. However, existing methods are restricted to learning from experiments sharing the same set of hyperparameters. In this paper, we introduce the OptFormer, the first text-based Transformer HPO framework that… ▽ More Meta-learning hyperparameter optimization (HPO) algorithms from prior experiments is a promising approach to improve optimization efficiency over objective functions from a similar distribution. However, existing methods are restricted to learning from experiments sharing the same set of hyperparameters. In this paper, we introduce the OptFormer, the first text-based Transformer HPO framework that provides a universal end-to-end interface for jointly learning policy and function prediction when trained on vast tuning data from the wild, such as Google's Vizier database, one of the world's largest HPO datasets. Our extensive experiments demonstrate that the OptFormer can simultaneously imitate at least 7 different HPO algorithms, which can be further improved via its function uncertainty estimates. Compared to a Gaussian Process, the OptFormer also learns a robust prior distribution for hyperparameter response functions, and can thereby provide more accurate and better calibrated predictions. This work paves the path to future extensions for training a Transformer-based model as a general HPO optimizer. △ Less

Submitted 13 October, 2022; v1 submitted 26 May, 2022; originally announced May 2022.

Comments: Published as a conference paper in Neural Information Processing Systems (NeurIPS) 2022. Code can be found in https://github.com/google-research/optformer and Google AI Blog can be found in https://ai.googleblog.com/2022/08/optformer-towards-universal.html