-
ConfEviSurrogate: A Conformalized Evidential Surrogate Model for Uncertainty Quantification
Authors:
Yuhan Duan,
Xin Zhao,
Neng Shi,
Han-Wei Shen
Abstract:
Surrogate models, crucial for approximating complex simulation data across sciences, inherently carry uncertainties that range from simulation noise to model prediction errors. Without rigorous uncertainty quantification, predictions become unreliable and hence hinder analysis. While methods like Monte Carlo dropout and ensemble models exist, they are often costly, fail to isolate uncertainty type…
▽ More
Surrogate models, crucial for approximating complex simulation data across sciences, inherently carry uncertainties that range from simulation noise to model prediction errors. Without rigorous uncertainty quantification, predictions become unreliable and hence hinder analysis. While methods like Monte Carlo dropout and ensemble models exist, they are often costly, fail to isolate uncertainty types, and lack guaranteed coverage in prediction intervals. To address this, we introduce ConfEviSurrogate, a novel Conformalized Evidential Surrogate Model that can efficiently learn high-order evidential distributions, directly predict simulation outcomes, separate uncertainty sources, and provide prediction intervals. A conformal prediction-based calibration step further enhances interval reliability to ensure coverage and improve efficiency. Our ConfEviSurrogate demonstrates accurate predictions and robust uncertainty estimates in diverse simulations, including cosmology, ocean dynamics, and fluid dynamics.
△ Less
Submitted 3 April, 2025;
originally announced April 2025.
-
Diffusion-Based Surrogate Modeling and Multi-Fidelity Calibration
Authors:
Naichen Shi,
Hao Yan,
Shenghan Guo,
Raed Al Kontar
Abstract:
Physics simulations have become fundamental tools to study myriad engineering systems. As physics simulations often involve simplifications, their outputs should be calibrated using real-world data. In this paper, we present a diffusion-based surrogate (DBS) that calibrates multi-fidelity physics simulations with diffusion generative processes. DBS categorizes multi-fidelity physics simulations in…
▽ More
Physics simulations have become fundamental tools to study myriad engineering systems. As physics simulations often involve simplifications, their outputs should be calibrated using real-world data. In this paper, we present a diffusion-based surrogate (DBS) that calibrates multi-fidelity physics simulations with diffusion generative processes. DBS categorizes multi-fidelity physics simulations into inexpensive and expensive simulations, depending on the computational costs. The inexpensive simulations, which can be obtained with low latency, directly inject contextual information into diffusion models. Furthermore, when results from expensive simulations are available, \name refines the quality of generated samples via a guided diffusion process. This design circumvents the need for large amounts of expensive physics simulations to train denoising diffusion models, thus lending flexibility to practitioners. DBS builds on Bayesian probabilistic models and is equipped with a theoretical guarantee that provides upper bounds on the Wasserstein distance between the sample and underlying true distribution. The probabilistic nature of DBS also provides a convenient approach for uncertainty quantification in prediction. Our models excel in cases where physics simulations are imperfect and sometimes inaccessible. We use a numerical simulation in fluid dynamics and a case study in laser-based metal powder deposition additive manufacturing to demonstrate how DBS calibrates multi-fidelity physics simulations with observations to obtain surrogates with superior predictive performance.
△ Less
Submitted 27 June, 2025; v1 submitted 24 July, 2024;
originally announced July 2024.
-
Personalized Tucker Decomposition: Modeling Commonality and Peculiarity on Tensor Data
Authors:
Jiuyun Hu,
Naichen Shi,
Raed Al Kontar,
Hao Yan
Abstract:
We propose personalized Tucker decomposition (perTucker) to address the limitations of traditional tensor decomposition methods in capturing heterogeneity across different datasets. perTucker decomposes tensor data into shared global components and personalized local components. We introduce a mode orthogonality assumption and develop a proximal gradient regularized block coordinate descent algori…
▽ More
We propose personalized Tucker decomposition (perTucker) to address the limitations of traditional tensor decomposition methods in capturing heterogeneity across different datasets. perTucker decomposes tensor data into shared global components and personalized local components. We introduce a mode orthogonality assumption and develop a proximal gradient regularized block coordinate descent algorithm that is guaranteed to converge to a stationary point. By learning unique and common representations across datasets, we demonstrate perTucker's effectiveness in anomaly detection, client classification, and clustering through a simulation study and two case studies on solar flare detection and tonnage signal classification.
△ Less
Submitted 6 September, 2023;
originally announced September 2023.
-
Heterogeneous Matrix Factorization: When Features Differ by Datasets
Authors:
Naichen Shi,
Raed Al Kontar,
Salar Fattahi
Abstract:
In myriad statistical applications, data are collected from related but heterogeneous sources. These sources share some commonalities while containing idiosyncratic characteristics. One of the most fundamental challenges in such scenarios is to recover the shared and source-specific factors. Despite the existence of a few heuristic approaches, a generic algorithm with theoretical guarantees has ye…
▽ More
In myriad statistical applications, data are collected from related but heterogeneous sources. These sources share some commonalities while containing idiosyncratic characteristics. One of the most fundamental challenges in such scenarios is to recover the shared and source-specific factors. Despite the existence of a few heuristic approaches, a generic algorithm with theoretical guarantees has yet to be established. In this paper, we tackle the problem by proposing a method called Heterogeneous Matrix Factorization to separate the shared and unique factors for a class of problems. HMF maintains the orthogonality between the shared and unique factors by leveraging an invariance property in the objective. The algorithm is easy to implement and intrinsically distributed. On the theoretic side, we show that for the square error loss, HMF will converge into the optimal solutions, which are close to the ground truth. HMF can be integrated auto-encoders to learn nonlinear feature mappings. Through a variety of case studies, we showcase HMF's benefits and applicability in video segmentation, time-series feature extraction, and recommender systems.
△ Less
Submitted 27 March, 2024; v1 submitted 28 May, 2023;
originally announced May 2023.
-
Personalized PCA: Decoupling Shared and Unique Features
Authors:
Naichen Shi,
Raed Al Kontar
Abstract:
In this paper, we tackle a significant challenge in PCA: heterogeneity. When data are collected from different sources with heterogeneous trends while still sharing some congruency, it is critical to extract shared knowledge while retaining the unique features of each source. To this end, we propose personalized PCA (PerPCA), which uses mutually orthogonal global and local principal components to…
▽ More
In this paper, we tackle a significant challenge in PCA: heterogeneity. When data are collected from different sources with heterogeneous trends while still sharing some congruency, it is critical to extract shared knowledge while retaining the unique features of each source. To this end, we propose personalized PCA (PerPCA), which uses mutually orthogonal global and local principal components to encode both unique and shared features. We show that, under mild conditions, both unique and shared features can be identified and recovered by a constrained optimization problem, even if the covariance matrices are immensely different. Also, we design a fully federated algorithm inspired by distributed Stiefel gradient descent to solve the problem. The algorithm introduces a new group of operations called generalized retractions to handle orthogonality constraints, and only requires global PCs to be shared across sources. We prove the linear convergence of the algorithm under suitable assumptions. Comprehensive numerical experiments highlight PerPCA's superior performance in feature extraction and prediction from heterogeneous datasets. As a systematic approach to decouple shared and unique features from heterogeneous datasets, PerPCA finds applications in several tasks, including video segmentation, topic extraction, and feature clustering.
△ Less
Submitted 8 February, 2024; v1 submitted 16 July, 2022;
originally announced July 2022.
-
Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning
Authors:
Naichen Shi,
Fan Lai,
Raed Al Kontar,
Mosharaf Chowdhury
Abstract:
In this paper we propose Fed-ensemble: a simple approach that bringsmodel ensembling to federated learning (FL). Instead of aggregating localmodels to update a single global model, Fed-ensemble uses random permutations to update a group of K models and then obtains predictions through model averaging. Fed-ensemble can be readily utilized within established FL methods and does not impose a computat…
▽ More
In this paper we propose Fed-ensemble: a simple approach that bringsmodel ensembling to federated learning (FL). Instead of aggregating localmodels to update a single global model, Fed-ensemble uses random permutations to update a group of K models and then obtains predictions through model averaging. Fed-ensemble can be readily utilized within established FL methods and does not impose a computational overhead as it only requires one of the K models to be sent to a client in each communication round. Theoretically, we show that predictions on newdata from all K models belong to the same predictive posterior distribution under a neural tangent kernel regime. This result in turn sheds light onthe generalization advantages of model averaging. We also illustrate thatFed-ensemble has an elegant Bayesian interpretation. Empirical results show that our model has superior performance over several FL algorithms,on a wide range of data sets, and excels in heterogeneous settings often encountered in FL applications.
△ Less
Submitted 21 July, 2021;
originally announced July 2021.
-
Sequential Gibbs Sampling Algorithm for Cognitive Diagnosis Models with Many Attributes
Authors:
Juntao Wang,
Ningzhong Shi,
Xue Zhang,
Gongjun Xu
Abstract:
Cognitive diagnosis models (CDMs) are useful statistical tools to provide rich information relevant for intervention and learning. As a popular approach to estimate and make inference of CDMs, the Markov chain Monte Carlo (MCMC) algorithm is widely used in practice. However, when the number of attributes, $K$, is large, the existing MCMC algorithm may become time-consuming, due to the fact that…
▽ More
Cognitive diagnosis models (CDMs) are useful statistical tools to provide rich information relevant for intervention and learning. As a popular approach to estimate and make inference of CDMs, the Markov chain Monte Carlo (MCMC) algorithm is widely used in practice. However, when the number of attributes, $K$, is large, the existing MCMC algorithm may become time-consuming, due to the fact that $O(2^K)$ calculations are usually needed in the process of MCMC sampling to get the conditional distribution for each attribute profile. To overcome this computational issue, motivated by Culpepper and Hudson (2018), we propose a computationally efficient sequential Gibbs sampling method, which needs $O(K)$ calculations to sample each attribute profile. We use simulation and real data examples to show the good finite-sample performance of the proposed sequential Gibbs sampling, and its advantage over existing methods.
△ Less
Submitted 13 February, 2021; v1 submitted 24 June, 2020;
originally announced June 2020.
-
Variable Selection Incorporating Prior Constraint Information into Lasso
Authors:
Shurong Zheng,
Guodong Song,
Ning-Zhong Shi
Abstract:
We propose the variable selection procedure incorporating prior constraint information into lasso. The proposed procedure combines the sample and prior information, and selects significant variables for responses in a narrower region where the true parameters lie. It increases the efficiency to choose the true model correctly. The proposed procedure can be executed by many constrained quadratic…
▽ More
We propose the variable selection procedure incorporating prior constraint information into lasso. The proposed procedure combines the sample and prior information, and selects significant variables for responses in a narrower region where the true parameters lie. It increases the efficiency to choose the true model correctly. The proposed procedure can be executed by many constrained quadratic programming methods and the initial estimator can be found by least square or Monte Carlo method. The proposed procedure also enjoys good theoretical properties. Moreover, the proposed procedure is not only used for linear models but also can be used for generalized linear models({\sl GLM}), Cox models, quantile regression models and many others with the help of Wang and Leng (2007)'s LSA, which changes these models as the approximation of linear models. The idea of combining sample and prior constraint information can be also used for other modified lasso procedures. Some examples are used for illustration of the idea of incorporating prior constraint information in variable selection procedures.
△ Less
Submitted 31 May, 2007;
originally announced May 2007.
