Showing 1–46 of 46 results for author: Wang, N

Conformal Prediction Adaptive to Unknown Subpopulation Shifts

Authors: Nien-Shao Wang, Duygu Nur Yaldiz, Yavuz Faruk Bakman, Sai Praneeth Karimireddy

Abstract: Conformal prediction is widely used to equip black-box machine learning models with uncertainty quantification enjoying formal coverage guarantees. However, these guarantees typically break down in the presence of distribution shifts, where the data distribution at test time differs from the training (or calibration-time) distribution. In this work, we address subpopulation shifts, where the test… ▽ More Conformal prediction is widely used to equip black-box machine learning models with uncertainty quantification enjoying formal coverage guarantees. However, these guarantees typically break down in the presence of distribution shifts, where the data distribution at test time differs from the training (or calibration-time) distribution. In this work, we address subpopulation shifts, where the test environment exhibits an unknown and differing mixture of subpopulations compared to the calibration data. We propose new methods that provably adapt conformal prediction to such shifts, ensuring valid coverage without requiring explicit knowledge of subpopulation structure. Our algorithms scale to high-dimensional settings and perform effectively in realistic machine learning tasks. Extensive experiments on vision (with vision transformers) and language (with large language models) benchmarks demonstrate that our methods reliably maintain coverage and controls risk in scenarios where standard conformal prediction fails. △ Less

Submitted 5 June, 2025; originally announced June 2025.

Comments: 20 pages, 6 figures, 5 tables, submitted to NeurIPS 2025

AI-Assisted Conversational Interviewing: Effects on Data Quality and User Experience

Authors: Soubhik Barari, Jarret Angbazo, Natalie Wang, Leah M. Christian, Elizabeth Dean, Zoe Slowinski, Brandon Sepulvado

Abstract: Standardized surveys scale efficiently but sacrifice depth, while conversational interviews improve response quality at the cost of scalability and consistency. This study bridges the gap between these methods by introducing a framework for AI-assisted conversational interviewing. To evaluate this framework, we conducted a web survey experiment where 1,800 participants were randomly assigned to te… ▽ More Standardized surveys scale efficiently but sacrifice depth, while conversational interviews improve response quality at the cost of scalability and consistency. This study bridges the gap between these methods by introducing a framework for AI-assisted conversational interviewing. To evaluate this framework, we conducted a web survey experiment where 1,800 participants were randomly assigned to text-based conversational AI agents, or "textbots", to dynamically probe respondents for elaboration and interactively code open-ended responses. We assessed textbot performance in terms of coding accuracy, response quality, and respondent experience. Our findings reveal that textbots perform moderately well in live coding even without survey-specific fine-tuning, despite slightly inflated false positive errors due to respondent acquiescence bias. Open-ended responses were more detailed and informative, but this came at a slight cost to respondent experience. Our findings highlight the feasibility of using AI methods to enhance open-ended data collection in web surveys. △ Less

Submitted 9 April, 2025; originally announced April 2025.

Cornering in the Water: An Investigation of Dolphin Swimming Performance

Authors: Mingkai Xia, Junhan Zhang, Ningshan Wang, Gabriel Antoniak, Nicole West, Ding Zhang, Kenneth Alex Shorter

Abstract: This article provides new insights into dolphin maneuver strategies in lap swimming tasks. However, most existing research focuses on straight-line swimming leaving the study of dolphins' corning strategies an open area. Challenges for directly analyzing dolphins' turning behavior include difficulties in motion tracking underwater and the inability to directly measure the propulsive forces. This p… ▽ More This article provides new insights into dolphin maneuver strategies in lap swimming tasks. However, most existing research focuses on straight-line swimming leaving the study of dolphins' corning strategies an open area. Challenges for directly analyzing dolphins' turning behavior include difficulties in motion tracking underwater and the inability to directly measure the propulsive forces. This paper provides methodology and analyses of dolphins' swimming performance during lap swimming tasks. External camera detection and internal kinematics measured from wearable bio-tags are involved in this study to support accurate localization of the animals. A particle filter, which fuses the external and internal measurements, is implemented to provide accurate estimations of the trajectories, even when they swim deep below the water's surface. Thereafter, a hydrodynamic model is constructed to calculate the thrust power and energy cost of the animals. The energetic cost during lap swimming is calculated for the comparison between different corning behaviors. The results show that the implemented particle filter can provide precise and complete trajectories of the tested dolphins, providing fundamental for statistical study of the corning behavior. From the kinematic analysis, TT01 is the fastest lap swimmer, with the highest swimming speed for the whole lap while performing a sharp turn with small deceleration. TT02 performs greater energetic efficiency than TT01 by transferring more weight at high speed. TT03 shows the highest energetic efficiency by maintaining a slow underwater motion. △ Less

Submitted 26 November, 2024; originally announced November 2024.

Comments: 11 pages, 13 figures

Deep Learning Framework for History Matching CO2 Storage with 4D Seismic and Monitoring Well Data

Authors: Nanzhe Wang, Louis J. Durlofsky

Abstract: Geological carbon storage entails the injection of megatonnes of supercritical CO2 into subsurface formations. The properties of these formations are usually highly uncertain, which makes design and optimization of large-scale storage operations challenging. In this paper we introduce a history matching strategy that enables the calibration of formation properties based on early-time observations.… ▽ More Geological carbon storage entails the injection of megatonnes of supercritical CO2 into subsurface formations. The properties of these formations are usually highly uncertain, which makes design and optimization of large-scale storage operations challenging. In this paper we introduce a history matching strategy that enables the calibration of formation properties based on early-time observations. Early-time assessments are essential to assure the operation is performing as planned. Our framework involves two fit-for-purpose deep learning surrogate models that provide predictions for in-situ monitoring well data and interpreted time-lapse (4D) seismic saturation data. These two types of data are at very different scales of resolution, so it is appropriate to construct separate, specialized deep learning networks for their prediction. This approach results in a workflow that is more straightforward to design and more efficient to train than a single surrogate that provides global high-fidelity predictions. The deep learning models are integrated into a hierarchical Markov chain Monte Carlo (MCMC) history matching procedure. History matching is performed on a synthetic case with and without 4D seismic data, which allows us to quantify the impact of 4D seismic on uncertainty reduction. The use of both data types is shown to provide substantial uncertainty reduction in key geomodel parameters and to enable accurate predictions of CO2 plume dynamics. The overall history matching framework developed in this study represents an efficient way to integrate multiple data types and to assess the impact of each on uncertainty reduction and performance predictions. △ Less

Submitted 21 January, 2025; v1 submitted 2 August, 2024; originally announced August 2024.

Comments: 52 pages, 21 figures

Evaluating the evolution and inter-individual variability of infant functional module development from 0 to 5 years old

Authors: Lingbin Bian, Nizhuan Wang, Yuanning Li, Adeel Razi, Qian Wang, Han Zhang, Dinggang Shen, the UNC/UMN Baby Connectome Project Consortium

Abstract: The segregation and integration of infant brain networks undergo tremendous changes due to the rapid development of brain function and organization. Traditional methods for estimating brain modularity usually rely on group-averaged functional connectivity (FC), often overlooking individual variability. To address this, we introduce a novel approach utilizing Bayesian modeling to analyze the dynami… ▽ More The segregation and integration of infant brain networks undergo tremendous changes due to the rapid development of brain function and organization. Traditional methods for estimating brain modularity usually rely on group-averaged functional connectivity (FC), often overlooking individual variability. To address this, we introduce a novel approach utilizing Bayesian modeling to analyze the dynamic development of functional modules in infants over time. This method retains inter-individual variability and, in comparison to conventional group averaging techniques, more effectively detects modules, taking into account the stationarity of module evolution. Furthermore, we explore gender differences in module development under awake and sleep conditions by assessing modular similarities. Our results show that female infants demonstrate more distinct modular structures between these two conditions, possibly implying relative quiet and restful sleep compared with male infants. △ Less

Submitted 17 July, 2024; originally announced July 2024.

Coordinated Sparse Recovery of Label Noise

Authors: Yukun Yang, Naihao Wang, Haixin Yang, Ruirui Li

Abstract: Label noise is a common issue in real-world datasets that inevitably impacts the generalization of models. This study focuses on robust classification tasks where the label noise is instance-dependent. Estimating the transition matrix accurately in this task is challenging, and methods based on sample selection often exhibit confirmation bias to varying degrees. Sparse over-parameterized training… ▽ More Label noise is a common issue in real-world datasets that inevitably impacts the generalization of models. This study focuses on robust classification tasks where the label noise is instance-dependent. Estimating the transition matrix accurately in this task is challenging, and methods based on sample selection often exhibit confirmation bias to varying degrees. Sparse over-parameterized training (SOP) has been theoretically effective in estimating and recovering label noise, offering a novel solution for noise-label learning. However, this study empirically observes and verifies a technical flaw of SOP: the lack of coordination between model predictions and noise recovery leads to increased generalization error. To address this, we propose a method called Coordinated Sparse Recovery (CSR). CSR introduces a collaboration matrix and confidence weights to coordinate model predictions and noise recovery, reducing error leakage. Based on CSR, this study designs a joint sample selection strategy and constructs a comprehensive and powerful learning framework called CSR+. CSR+ significantly reduces confirmation bias, especially for datasets with more classes and a high proportion of instance-specific noise. Experimental results on simulated and real-world noisy datasets demonstrate that both CSR and CSR+ achieve outstanding performance compared to methods at the same level. △ Less

Submitted 6 April, 2024; originally announced April 2024.

Comments: Pre-print prior to submission to journal

High-dimensional Tensor Response Regression using the t-Distribution

Authors: Ning Wang, Xin Zhang, Qing Mai

Abstract: In recent years, promising statistical modeling approaches to tensor data analysis have been rapidly developed. Traditional multivariate analysis tools, such as multivariate regression and discriminant analysis, are generalized from modeling random vectors and matrices to higher-order random tensors. One of the biggest challenges to statistical tensor models is the non-Gaussian nature of many real… ▽ More In recent years, promising statistical modeling approaches to tensor data analysis have been rapidly developed. Traditional multivariate analysis tools, such as multivariate regression and discriminant analysis, are generalized from modeling random vectors and matrices to higher-order random tensors. One of the biggest challenges to statistical tensor models is the non-Gaussian nature of many real-world data. Unfortunately, existing approaches are either restricted to normality or implicitly using least squares type objective functions that are computationally efficient but sensitive to data contamination. Motivated by this, we adopt a simple tensor t-distribution that is, unlike the commonly used matrix t-distributions, compatible with tensor operators and reshaping of the data. We study the tensor response regression with tensor t-error, and develop penalized likelihood-based estimation and a novel one-step estimation. We study the asymptotic relative efficiency of various estimators and establish the one-step estimator's oracle properties and near-optimal asymptotic efficiency. We further propose a high-dimensional modification to the one-step estimation procedure and show that it attains the minimax optimal rate in estimation. Numerical studies show the excellent performance of the one-step estimator. △ Less

Submitted 21 June, 2023; originally announced June 2023.

Hierarchical Bayesian inference for community detection and connectivity of functional brain networks

Authors: Lingbin Bian, Nizhuan Wang, Leonardo Novelli, Jonathan Keith, Adeel Razi

Abstract: Many functional magnetic resonance imaging (fMRI) studies rely on estimates of hierarchically organised brain networks whose segregation and integration reflect the dynamic transitions of latent cognitive states. However, most existing methods for estimating the community structure of networks from both individual and group-level analysis neglect the variability between subjects and lack validatio… ▽ More Many functional magnetic resonance imaging (fMRI) studies rely on estimates of hierarchically organised brain networks whose segregation and integration reflect the dynamic transitions of latent cognitive states. However, most existing methods for estimating the community structure of networks from both individual and group-level analysis neglect the variability between subjects and lack validation. In this paper, we develop a new multilayer community detection method based on Bayesian latent block modelling. The method can robustly detect the group-level community structure of weighted functional networks that give rise to hidden brain states with an unknown number of communities and retain the variability of individual networks. For validation, we propose a new community structure-based multivariate Gaussian generative model to simulate synthetic signal. Our result shows that the inferred community memberships using hierarchical Bayesian analysis are consistent with the predefined node labels in the generative model. The method is also tested using real working memory task-fMRI data of 100 unrelated healthy subjects from the Human Connectome Project. The results show distinctive community structure patterns between 2-back, 0-back, and fixation conditions, which may reflect cognitive and behavioural states under working memory task conditions. △ Less

Submitted 26 May, 2024; v1 submitted 18 January, 2023; originally announced January 2023.

Plateau in Monotonic Linear Interpolation -- A "Biased" View of Loss Landscape for Deep Networks

Authors: Xiang Wang, Annie N. Wang, Mo Zhou, Rong Ge

Abstract: Monotonic linear interpolation (MLI) - on the line connecting a random initialization with the minimizer it converges to, the loss and accuracy are monotonic - is a phenomenon that is commonly observed in the training of neural networks. Such a phenomenon may seem to suggest that optimization of neural networks is easy. In this paper, we show that the MLI property is not necessarily related to the… ▽ More Monotonic linear interpolation (MLI) - on the line connecting a random initialization with the minimizer it converges to, the loss and accuracy are monotonic - is a phenomenon that is commonly observed in the training of neural networks. Such a phenomenon may seem to suggest that optimization of neural networks is easy. In this paper, we show that the MLI property is not necessarily related to the hardness of optimization problems, and empirical observations on MLI for deep neural networks depend heavily on biases. In particular, we show that interpolating both weights and biases linearly leads to very different influences on the final output, and when different classes have different last-layer biases on a deep network, there will be a long plateau in both the loss and accuracy interpolation (which existing theory of MLI cannot explain). We also show how the last-layer biases for different classes can be different even on a perfectly balanced dataset using a simple model. Empirically we demonstrate that similar intuitions hold on practical networks and realistic datasets. △ Less

Submitted 14 February, 2023; v1 submitted 3 October, 2022; originally announced October 2022.

Comments: ICLR 2023

Deep-learning-based upscaling method for geologic models via theory-guided convolutional neural network

Authors: Nanzhe Wang, Qinzhuo Liao, Haibin Chang, Dongxiao Zhang

Abstract: Large-scale or high-resolution geologic models usually comprise a huge number of grid blocks, which can be computationally demanding and time-consuming to solve with numerical simulators. Therefore, it is advantageous to upscale geologic models (e.g., hydraulic conductivity) from fine-scale (high-resolution grids) to coarse-scale systems. Numerical upscaling methods have been proven to be effectiv… ▽ More Large-scale or high-resolution geologic models usually comprise a huge number of grid blocks, which can be computationally demanding and time-consuming to solve with numerical simulators. Therefore, it is advantageous to upscale geologic models (e.g., hydraulic conductivity) from fine-scale (high-resolution grids) to coarse-scale systems. Numerical upscaling methods have been proven to be effective and robust for coarsening geologic models, but their efficiency remains to be improved. In this work, a deep-learning-based method is proposed to upscale the fine-scale geologic models, which can assist to improve upscaling efficiency significantly. In the deep learning method, a deep convolutional neural network (CNN) is trained to approximate the relationship between the coarse grid of hydraulic conductivity fields and the hydraulic heads, which can then be utilized to replace the numerical solvers while solving the flow equations for each coarse block. In addition, physical laws (e.g., governing equations and periodic boundary conditions) can also be incorporated into the training process of the deep CNN model, which is termed the theory-guided convolutional neural network (TgCNN). With the physical information considered, dependence on the data volume of training the deep learning models can be reduced greatly. Several subsurface flow cases are introduced to test the performance of the proposed deep-learning-based upscaling method, including 2D and 3D cases, and isotropic and anisotropic cases. The results show that the deep learning method can provide equivalent upscaling accuracy to the numerical method, and efficiency can be improved significantly compared to numerical upscaling. △ Less

Submitted 31 December, 2021; originally announced January 2022.

Comments: 37 pages, 21 pages

Kernel Mean Estimation by Marginalized Corrupted Distributions

Authors: Xiaobo Xia, Shuo Shan, Mingming Gong, Nannan Wang, Fei Gao, Haikun Wei, Tongliang Liu

Abstract: Estimating the kernel mean in a reproducing kernel Hilbert space is a critical component in many kernel learning algorithms. Given a finite sample, the standard estimate of the target kernel mean is the empirical average. Previous works have shown that better estimators can be constructed by shrinkage methods. In this work, we propose to corrupt data examples with noise from known distributions an… ▽ More Estimating the kernel mean in a reproducing kernel Hilbert space is a critical component in many kernel learning algorithms. Given a finite sample, the standard estimate of the target kernel mean is the empirical average. Previous works have shown that better estimators can be constructed by shrinkage methods. In this work, we propose to corrupt data examples with noise from known distributions and present a new kernel mean estimator, called the marginalized kernel mean estimator, which estimates kernel mean under the corrupted distribution. Theoretically, we show that the marginalized kernel mean estimator introduces implicit regularization in kernel mean estimation. Empirically, we show on a variety of datasets that the marginalized kernel mean estimator obtains much lower estimation error than the existing estimators. △ Less

Submitted 10 July, 2021; originally announced July 2021.

CORe: Capitalizing On Rewards in Bandit Exploration

Authors: Nan Wang, Branislav Kveton, Maryam Karimzadehgan

Abstract: We propose a bandit algorithm that explores purely by randomizing its past observations. In particular, the sufficient optimism in the mean reward estimates is achieved by exploiting the variance in the past observed rewards. We name the algorithm Capitalizing On Rewards (CORe). The algorithm is general and can be easily applied to different bandit settings. The main benefit of CORe is that its ex… ▽ More We propose a bandit algorithm that explores purely by randomizing its past observations. In particular, the sufficient optimism in the mean reward estimates is achieved by exploiting the variance in the past observed rewards. We name the algorithm Capitalizing On Rewards (CORe). The algorithm is general and can be easily applied to different bandit settings. The main benefit of CORe is that its exploration is fully data-dependent. It does not rely on any external noise and adapts to different problems without parameter tuning. We derive a $\tilde O(d\sqrt{n\log K})$ gap-free bound on the $n$-round regret of CORe in a stochastic linear bandit, where $d$ is the number of features and $K$ is the number of arms. Extensive empirical evaluation on multiple synthetic and real-world problems demonstrates the effectiveness of CORe. △ Less

Submitted 7 March, 2021; originally announced March 2021.

Multiscale Comparative Connectomics

Authors: Vivek Gopalakrishnan, Jaewon Chung, Eric Bridgeford, Benjamin D. Pedigo, Jesús Arroyo, Lucy Upchurch, G. Allan Johnson, Nian Wang, Youngser Park, Carey E. Priebe, Joshua T. Vogelstein

Abstract: The connectome, a map of the structural and/or functional connections in the brain, provides a complex representation of the neurobiological phenotypes on which it supervenes. This information-rich data modality has the potential to transform our understanding of the relationship between patterns in brain connectivity and neurological processes, disorders, and diseases. However, existing computati… ▽ More The connectome, a map of the structural and/or functional connections in the brain, provides a complex representation of the neurobiological phenotypes on which it supervenes. This information-rich data modality has the potential to transform our understanding of the relationship between patterns in brain connectivity and neurological processes, disorders, and diseases. However, existing computational techniques used to analyze connectomes are oftentimes insufficient for interrogating multi-subject connectomics datasets: many current methods are either solely designed to analyze single connectomes or leverage heuristic graph statistics that are unable to capture the complete topology of multiscale connections between brain regions. To enable more rigorous connectomics analysis, we introduce a set of robust and interpretable effect size measures motivated by recent theoretical advances in random graph models. These measures facilitate simultaneous analysis of multiple connectomes across different scales of network topology, enabling the robust and reproducible discovery of hierarchical brain structures that vary in relation to phenotypic profiles. In addition to explaining the theoretical foundations and guarantees of our algorithms, we demonstrate their superiority over current state-of-the-art connectomics methods through extensive simulation studies and real-data experiments. Using a set of high-resolution connectomes obtained from genetically distinct mouse strains (including the BTBR mouse -- a standard model of autism -- and three behavioral wild-types), we illustrate how our methods successfully uncover latent information in multi-subject connectomics data and yield valuable insights into the connective correlates of neurological phenotypes that other methods do not capture. The data and code necessary to reproduce our analyses are available at https://github.com/neurodata/MCC. △ Less

Submitted 2 December, 2024; v1 submitted 30 November, 2020; originally announced November 2020.

Deep-learning based discovery of partial differential equations in integral form from sparse and noisy data

Authors: Hao Xu, Dongxiao Zhang, Nanzhe Wang

Abstract: Data-driven discovery of partial differential equations (PDEs) has attracted increasing attention in recent years. Although significant progress has been made, certain unresolved issues remain. For example, for PDEs with high-order derivatives, the performance of existing methods is unsatisfactory, especially when the data are sparse and noisy. It is also difficult to discover heterogeneous parame… ▽ More Data-driven discovery of partial differential equations (PDEs) has attracted increasing attention in recent years. Although significant progress has been made, certain unresolved issues remain. For example, for PDEs with high-order derivatives, the performance of existing methods is unsatisfactory, especially when the data are sparse and noisy. It is also difficult to discover heterogeneous parametric PDEs where heterogeneous parameters are embedded in the partial differential operators. In this work, a new framework combining deep-learning and integral form is proposed to handle the above-mentioned problems simultaneously, and improve the accuracy and stability of PDE discovery. In the framework, a deep neural network is firstly trained with observation data to generate meta-data and calculate derivatives. Then, a unified integral form is defined, and the genetic algorithm is employed to discover the best structure. Finally, the value of parameters is calculated, and whether the parameters are constants or variables is identified. Numerical experiments proved that our proposed algorithm is more robust to noise and more accurate compared with existing methods due to the utilization of integral form. Our proposed algorithm is also able to discover PDEs with high-order derivatives or heterogeneous parameters accurately with sparse and noisy data. △ Less

Submitted 24 November, 2020; originally announced November 2020.

Journal ref: Journal of Computational Physics, 445 (2021), 110592

Theory-guided Auto-Encoder for Surrogate Construction and Inverse Modeling

Authors: Nanzhe Wang, Haibin Chang, Dongxiao Zhang

Abstract: A Theory-guided Auto-Encoder (TgAE) framework is proposed for surrogate construction and is further used for uncertainty quantification and inverse modeling tasks. The framework is built based on the Auto-Encoder (or Encoder-Decoder) architecture of convolutional neural network (CNN) via a theory-guided training process. In order to achieve the theory-guided training, the governing equations of th… ▽ More A Theory-guided Auto-Encoder (TgAE) framework is proposed for surrogate construction and is further used for uncertainty quantification and inverse modeling tasks. The framework is built based on the Auto-Encoder (or Encoder-Decoder) architecture of convolutional neural network (CNN) via a theory-guided training process. In order to achieve the theory-guided training, the governing equations of the studied problems can be discretized and the finite difference scheme of the equations can be embedded into the training of CNN. The residual of the discretized governing equations as well as the data mismatch constitute the loss function of the TgAE. The trained TgAE can be used to construct a surrogate that approximates the relationship between the model parameters and responses with limited labeled data. In order to test the performance of the TgAE, several subsurface flow cases are introduced. The results show the satisfactory accuracy of the TgAE surrogate and efficiency of uncertainty quantification tasks can be improved with the TgAE surrogate. The TgAE also shows good extrapolation ability for cases with different correlation lengths and variances. Furthermore, the parameter inversion task has been implemented with the TgAE surrogate and satisfactory results can be obtained. △ Less

Submitted 17 November, 2020; originally announced November 2020.

Journal ref: Comput. Methods Appl. Mech. Engrg., 385 (2021), 114037

A Lagrangian Dual-based Theory-guided Deep Neural Network

Authors: Miao Rong, Dongxiao Zhang, Nanzhe Wang

Abstract: The theory-guided neural network (TgNN) is a kind of method which improves the effectiveness and efficiency of neural network architectures by incorporating scientific knowledge or physical information. Despite its great success, the theory-guided (deep) neural network possesses certain limits when maintaining a tradeoff between training data and domain knowledge during the training process. In th… ▽ More The theory-guided neural network (TgNN) is a kind of method which improves the effectiveness and efficiency of neural network architectures by incorporating scientific knowledge or physical information. Despite its great success, the theory-guided (deep) neural network possesses certain limits when maintaining a tradeoff between training data and domain knowledge during the training process. In this paper, the Lagrangian dual-based TgNN (TgNN-LD) is proposed to improve the effectiveness of TgNN. We convert the original loss function into a constrained form with fewer items, in which partial differential equations (PDEs), engineering controls (ECs), and expert knowledge (EK) are regarded as constraints, with one Lagrangian variable per constraint. These Lagrangian variables are incorporated to achieve an equitable tradeoff between observation data and corresponding constraints, in order to improve prediction accuracy, and conserve time and computational resources adjusted by an ad-hoc procedure. To investigate the performance of the proposed method, the original TgNN model with a set of optimized weight values adjusted by ad-hoc procedures is compared on a subsurface flow problem, with their L2 error, R square (R2), and computational time being analyzed. Experimental results demonstrate the superiority of the Lagrangian dual-based TgNN. △ Less

Submitted 23 August, 2020; originally announced August 2020.

Comments: 12 pages, 10 figures

Journal ref: Complex & Intelligent Systems, 2022

Deep-Learning based Inverse Modeling Approaches: A Subsurface Flow Example

Authors: Nanzhe Wang, Haibin Chang, Dongxiao Zhang

Abstract: Deep-learning has achieved good performance and shown great potential for solving forward and inverse problems. In this work, two categories of innovative deep-learning based inverse modeling methods are proposed and compared. The first category is deep-learning surrogate-based inversion methods, in which the Theory-guided Neural Network (TgNN) is constructed as a deep-learning surrogate for probl… ▽ More Deep-learning has achieved good performance and shown great potential for solving forward and inverse problems. In this work, two categories of innovative deep-learning based inverse modeling methods are proposed and compared. The first category is deep-learning surrogate-based inversion methods, in which the Theory-guided Neural Network (TgNN) is constructed as a deep-learning surrogate for problems with uncertain model parameters. By incorporating physical laws and other constraints, the TgNN surrogate can be constructed with limited simulation runs and accelerate the inversion process significantly. Three TgNN surrogate-based inversion methods are proposed, including the gradient method, the iterative ensemble smoother (IES), and the training method. The second category is direct-deep-learning-inversion methods, in which TgNN constrained with geostatistical information, named TgNN-geo, is proposed for direct inverse modeling. In TgNN-geo, two neural networks are introduced to approximate the respective random model parameters and the solution. Since the prior geostatistical information can be incorporated, the direct-inversion method based on TgNN-geo works well, even in cases with sparse spatial measurements or imprecise prior statistics. Although the proposed deep-learning based inverse modeling methods are general in nature, and thus applicable to a wide variety of problems, they are tested with several subsurface flow problems. It is found that satisfactory results are obtained with a high efficiency. Moreover, both the advantages and disadvantages are further analyzed for the proposed two categories of deep-learning based inversion methods. △ Less

Submitted 28 July, 2020; originally announced July 2020.

Comments: 53 pages, 22 figures, 7 tables

Journal ref: Journal of Geophysical Research: Solid Earth, e2020JB020549, 2020

Space-time clustering of flash floods in a changing climate (China, 1950-2015)

Authors: Nan Wang, Luigi Lombardo, Marj Tonini, Weiming Cheng, Liang Guo, Junnan Xiong

Abstract: The persistence over space and time of flash flood disasters -- flash floods that have caused either economical or life losses, or both -- is a diagnostic measure of areas subjected to hydrological risk. The concept of persistence can be assessed via clustering analyses, performed here to analyse the national inventory of flash floods disasters in China occurred in the period 1950-2015. Specifical… ▽ More The persistence over space and time of flash flood disasters -- flash floods that have caused either economical or life losses, or both -- is a diagnostic measure of areas subjected to hydrological risk. The concept of persistence can be assessed via clustering analyses, performed here to analyse the national inventory of flash floods disasters in China occurred in the period 1950-2015. Specifically, we investigated the spatio-temporal pattern distribution of the flash floods and their clustering behavior by using both global and local methods: the first, based on the Ripley's K-function, and the second on scan statistics. As a result, we could visualize patterns of aggregated events, estimate the cluster duration and make assumptions about their evolution over time, also with respect precipitations trend. Due to the large spatial (the whole Chinese territory) and temporal scale of the dataset (66 years), we were able to capture whether certain clusters gather in specific locations and times, but also whether their magnitude tends to increase or decrease. Overall, the eastern regions in China are much more subjected to flash floods compared to the rest of the country. Detected clusters revealed that these phenomena predominantly occur between July and October, a period coinciding with the wet season in China. The number of detected clusters increases with time, but the associated duration drastically decreases in the recent period. This may indicate a change towards triggering mechanisms which are typical of short-duration extreme rainfall events. Finally, being flash floods directly linked to precipitation and their extreme realization, we indirectly assessed whether the magnitude of the trigger itself has also varied through space and time, enabling considerations in the context of climatic changes. △ Less

Submitted 23 June, 2020; originally announced June 2020.

Directional Multivariate Ranking

Authors: Nan Wang, Hongning Wang

Abstract: User-provided multi-aspect evaluations manifest users' detailed feedback on the recommended items and enable fine-grained understanding of their preferences. Extensive studies have shown that modeling such data greatly improves the effectiveness and explainability of the recommendations. However, as ranking is essential in recommendation, there is no principled solution yet for collectively genera… ▽ More User-provided multi-aspect evaluations manifest users' detailed feedback on the recommended items and enable fine-grained understanding of their preferences. Extensive studies have shown that modeling such data greatly improves the effectiveness and explainability of the recommendations. However, as ranking is essential in recommendation, there is no principled solution yet for collectively generating multiple item rankings over different aspects. In this work, we propose a directional multi-aspect ranking criterion to enable a holistic ranking of items with respect to multiple aspects. Specifically, we view multi-aspect evaluation as an integral effort from a user that forms a vector of his/her preferences over aspects. Our key insight is that the direction of the difference vector between two multi-aspect preference vectors reveals the pairwise order of comparison. Hence, it is necessary for a multi-aspect ranking criterion to preserve the observed directions from such pairwise comparisons. We further derive a complete solution for the multi-aspect ranking problem based on a probabilistic multivariate tensor factorization model. Comprehensive experimental analysis on a large TripAdvisor multi-aspect rating dataset and a Yelp review text dataset confirms the effectiveness of our solution. △ Less

Submitted 9 June, 2020; originally announced June 2020.

Comments: Accepted as a full research paper in KDD'20

Part-dependent Label Noise: Towards Instance-dependent Label Noise

Authors: Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, Masashi Sugiyama

Abstract: Learning with the \textit{instance-dependent} label noise is challenging, because it is hard to model such real-world noise. Note that there are psychological and physiological evidences showing that we humans perceive instances by decomposing them into parts. Annotators are therefore more likely to annotate instances based on the parts rather than the whole instances, where a wrong mapping from p… ▽ More Learning with the \textit{instance-dependent} label noise is challenging, because it is hard to model such real-world noise. Note that there are psychological and physiological evidences showing that we humans perceive instances by decomposing them into parts. Annotators are therefore more likely to annotate instances based on the parts rather than the whole instances, where a wrong mapping from parts to classes may cause the instance-dependent label noise. Motivated by this human cognition, in this paper, we approximate the instance-dependent label noise by exploiting \textit{part-dependent} label noise. Specifically, since instances can be approximately reconstructed by a combination of parts, we approximate the instance-dependent \textit{transition matrix} for an instance by a combination of the transition matrices for the parts of the instance. The transition matrices for parts can be learned by exploiting anchor points (i.e., data points that belong to a specific class almost surely). Empirical evaluations on synthetic and real-world datasets demonstrate our method is superior to the state-of-the-art approaches for learning from the instance-dependent label noise. △ Less

Submitted 2 December, 2020; v1 submitted 14 June, 2020; originally announced June 2020.

Class2Simi: A Noise Reduction Perspective on Learning with Noisy Labels

Authors: Songhua Wu, Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Nannan Wang, Haifeng Liu, Gang Niu

Abstract: Learning with noisy labels has attracted a lot of attention in recent years, where the mainstream approaches are in pointwise manners. Meanwhile, pairwise manners have shown great potential in supervised metric learning and unsupervised contrastive learning. Thus, a natural question is raised: does learning in a pairwise manner mitigate label noise? To give an affirmative answer, in this paper, we… ▽ More Learning with noisy labels has attracted a lot of attention in recent years, where the mainstream approaches are in pointwise manners. Meanwhile, pairwise manners have shown great potential in supervised metric learning and unsupervised contrastive learning. Thus, a natural question is raised: does learning in a pairwise manner mitigate label noise? To give an affirmative answer, in this paper, we propose a framework called Class2Simi: it transforms data points with noisy class labels to data pairs with noisy similarity labels, where a similarity label denotes whether a pair shares the class label or not. Through this transformation, the reduction of the noise rate is theoretically guaranteed, and hence it is in principle easier to handle noisy similarity labels. Amazingly, DNNs that predict the clean class labels can be trained from noisy data pairs if they are first pretrained from noisy data points. Class2Simi is computationally efficient because not only this transformation is on-the-fly in mini-batches, but also it just changes loss computation on top of model prediction into a pairwise manner. Its effectiveness is verified by extensive experiments. △ Less

Submitted 17 June, 2021; v1 submitted 14 June, 2020; originally announced June 2020.

The scalable Birth-Death MCMC Algorithm for Mixed Graphical Model Learning with Application to Genomic Data Integration

Authors: Nanwei Wang, Laurent Briollais, Helene Massam

Abstract: Recent advances in biological research have seen the emergence of high-throughput technologies with numerous applications that allow the study of biological mechanisms at an unprecedented depth and scale. A large amount of genomic data is now distributed through consortia like The Cancer Genome Atlas (TCGA), where specific types of biological information on specific type of tissue or cell are avai… ▽ More Recent advances in biological research have seen the emergence of high-throughput technologies with numerous applications that allow the study of biological mechanisms at an unprecedented depth and scale. A large amount of genomic data is now distributed through consortia like The Cancer Genome Atlas (TCGA), where specific types of biological information on specific type of tissue or cell are available. In cancer research, the challenge is now to perform integrative analyses of high-dimensional multi-omic data with the goal to better understand genomic processes that correlate with cancer outcomes, e.g. elucidate gene networks that discriminate a specific cancer subgroups (cancer sub-typing) or discovering gene networks that overlap across different cancer types (pan-cancer studies). In this paper, we propose a novel mixed graphical model approach to analyze multi-omic data of different types (continuous, discrete and count) and perform model selection by extending the Birth-Death MCMC (BDMCMC) algorithm initially proposed by \citet{stephens2000bayesian} and later developed by \cite{mohammadi2015bayesian}. We compare the performance of our method to the LASSO method and the standard BDMCMC method using simulations and find that our method is superior in terms of both computational efficiency and the accuracy of the model selection results. Finally, an application to the TCGA breast cancer data shows that integrating genomic information at different levels (mutation and expression data) leads to better subtyping of breast cancers. △ Less

Submitted 8 May, 2020; originally announced May 2020.

Efficient Uncertainty Quantification for Dynamic Subsurface Flow with Surrogate by Theory-guided Neural Network

Authors: Nanzhe Wang, Haibin Chang, Dongxiao Zhang

Abstract: Subsurface flow problems usually involve some degree of uncertainty. Consequently, uncertainty quantification is commonly necessary for subsurface flow prediction. In this work, we propose a methodology for efficient uncertainty quantification for dynamic subsurface flow with a surrogate constructed by the Theory-guided Neural Network (TgNN). The TgNN here is specially designed for problems with s… ▽ More Subsurface flow problems usually involve some degree of uncertainty. Consequently, uncertainty quantification is commonly necessary for subsurface flow prediction. In this work, we propose a methodology for efficient uncertainty quantification for dynamic subsurface flow with a surrogate constructed by the Theory-guided Neural Network (TgNN). The TgNN here is specially designed for problems with stochastic parameters. In the TgNN, stochastic parameters, time and location comprise the input of the neural network, while the quantity of interest is the output. The neural network is trained with available simulation data, while being simultaneously guided by theory (e.g., the governing equation, boundary conditions, initial conditions, etc.) of the underlying problem. The trained neural network can predict solutions of subsurface flow problems with new stochastic parameters. With the TgNN surrogate, the Monte Carlo (MC) method can be efficiently implemented for uncertainty quantification. The proposed methodology is evaluated with two-dimensional dynamic saturated flow problems in porous medium. Numerical results show that the TgNN based surrogate can significantly improve the efficiency of uncertainty quantification tasks compared with simulation based implementation. Further investigations regarding stochastic fields with smaller correlation length, larger variance, changing boundary values and out-of-distribution variances are performed, and satisfactory results are obtained. △ Less

Submitted 25 April, 2020; originally announced April 2020.

Penalized composite likelihood for colored graphical Gaussian models

Authors: Qiong Li, Xiaoying Sun, Nanwei Wang

Abstract: This paper proposes a penalized composite likelihood method for model selection in colored graphical Gaussian models. The method provides a sparse and symmetry-constrained estimator of the precision matrix, and thus conducts model selection and precision matrix estimation simultaneously. In particular, the method uses penalty terms to constrain the elements of the precision matrix, which enables u… ▽ More This paper proposes a penalized composite likelihood method for model selection in colored graphical Gaussian models. The method provides a sparse and symmetry-constrained estimator of the precision matrix, and thus conducts model selection and precision matrix estimation simultaneously. In particular, the method uses penalty terms to constrain the elements of the precision matrix, which enables us to transform the model selection problem into a constrained optimization problem. Further, computer experiments are conducted to illustrate the performance of the proposed new methodology. It is shown that the proposed method performs well in both the selection of nonzero elements in the precision matrix and the identification of symmetry structures in graphical models. The feasibility and potential clinical application of the proposed method are demonstrated on a microarray gene expression data set. △ Less

Submitted 2 April, 2020; originally announced April 2020.

MSC Class: 62H12 (Primary); 62F15 (Secondary)

Multi-Class Classification from Noisy-Similarity-Labeled Data

Authors: Songhua Wu, Xiaobo Xia, Tongliang Liu, Bo Han, Mingming Gong, Nannan Wang, Haifeng Liu, Gang Niu

Abstract: A similarity label indicates whether two instances belong to the same class while a class label shows the class of the instance. Without class labels, a multi-class classifier could be learned from similarity-labeled pairwise data by meta classification learning. However, since the similarity label is less informative than the class label, it is more likely to be noisy. Deep neural networks can ea… ▽ More A similarity label indicates whether two instances belong to the same class while a class label shows the class of the instance. Without class labels, a multi-class classifier could be learned from similarity-labeled pairwise data by meta classification learning. However, since the similarity label is less informative than the class label, it is more likely to be noisy. Deep neural networks can easily remember noisy data, leading to overfitting in classification. In this paper, we propose a method for learning from only noisy-similarity-labeled data. Specifically, to model the noise, we employ a noise transition matrix to bridge the class-posterior probability between clean and noisy data. We further estimate the transition matrix from only noisy data and build a novel learning system to learn a classifier which can assign noise-free class labels for instances. Moreover, we theoretically justify how our proposed method generalizes for learning classifiers. Experimental results demonstrate the superiority of the proposed method over the state-of-the-art method on benchmark-simulated and real-world noisy-label datasets. △ Less

Submitted 16 February, 2020; originally announced February 2020.

Deep Learning of Subsurface Flow via Theory-guided Neural Network

Authors: Nanzhe Wang, Dongxiao Zhang, Haibin Chang, Heng Li

Abstract: Active researches are currently being performed to incorporate the wealth of scientific knowledge into data-driven approaches (e.g., neural networks) in order to improve the latter's effectiveness. In this study, the Theory-guided Neural Network (TgNN) is proposed for deep learning of subsurface flow. In the TgNN, as supervised learning, the neural network is trained with available observations or… ▽ More Active researches are currently being performed to incorporate the wealth of scientific knowledge into data-driven approaches (e.g., neural networks) in order to improve the latter's effectiveness. In this study, the Theory-guided Neural Network (TgNN) is proposed for deep learning of subsurface flow. In the TgNN, as supervised learning, the neural network is trained with available observations or simulation data while being simultaneously guided by theory (e.g., governing equations, other physical constraints, engineering controls, and expert knowledge) of the underlying problem. The TgNN can achieve higher accuracy than the ordinary Artificial Neural Network (ANN) because the former provides physically feasible predictions and can be more readily generalized beyond the regimes covered with the training data. Furthermore, the TgNN model is proposed for subsurface flow with heterogeneous model parameters. Several numerical cases of two-dimensional transient saturated flow are introduced to test the performance of the TgNN. In the learning process, the loss function contains data mismatch, as well as PDE constraint, engineering control, and expert knowledge. After obtaining the parameters of the neural network by minimizing the loss function, a TgNN model is built that not only fits the data, but also adheres to physical/engineering constraints. Predicting the future response can be easily realized by the TgNN model. In addition, the TgNN model is tested in more complicated scenarios, such as prediction with changed boundary conditions, learning from noisy data or outliers, transfer learning, and engineering controls. Numerical results demonstrate that the TgNN model achieves much better predictability, reliability, and generalizability than ANN models due to the physical/engineering constraints in the former. △ Less

Submitted 24 October, 2019; originally announced November 2019.

Journal ref: Journal of Hydrology, 2020, 584, 124700

A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection

Authors: Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, Bingsheng He

Abstract: Federated learning has been a hot research topic in enabling the collaborative training of machine learning models among different organizations under the privacy restrictions. As researchers try to support more machine learning models with different privacy-preserving approaches, there is a requirement in developing systems and infrastructures to ease the development of various federated learning… ▽ More Federated learning has been a hot research topic in enabling the collaborative training of machine learning models among different organizations under the privacy restrictions. As researchers try to support more machine learning models with different privacy-preserving approaches, there is a requirement in developing systems and infrastructures to ease the development of various federated learning algorithms. Similar to deep learning systems such as PyTorch and TensorFlow that boost the development of deep learning, federated learning systems (FLSs) are equivalently important, and face challenges from various aspects such as effectiveness, efficiency, and privacy. In this survey, we conduct a comprehensive review on federated learning systems. To achieve smooth flow and guide future research, we introduce the definition of federated learning systems and analyze the system components. Moreover, we provide a thorough categorization for federated learning systems according to six different aspects, including data distribution, machine learning model, privacy mechanism, communication architecture, scale of federation and motivation of federation. The categorization can help the design of federated learning systems as shown in our case studies. By systematically summarizing the existing federated learning systems, we present the design factors, case studies, and future research opportunities. △ Less

Submitted 4 December, 2021; v1 submitted 23 July, 2019; originally announced July 2019.

Comments: Accepted to IEEE Transactions on Knowledge and Data Engineering (TKDE)

The FacT: Taming Latent Factor Models for Explainability with Factorization Trees

Authors: Yiyi Tao, Yiling Jia, Nan Wang, Hongning Wang

Abstract: Latent factor models have achieved great success in personalized recommendations, but they are also notoriously difficult to explain. In this work, we integrate regression trees to guide the learning of latent factor models for recommendation, and use the learnt tree structure to explain the resulting latent factors. Specifically, we build regression trees on users and items respectively with user… ▽ More Latent factor models have achieved great success in personalized recommendations, but they are also notoriously difficult to explain. In this work, we integrate regression trees to guide the learning of latent factor models for recommendation, and use the learnt tree structure to explain the resulting latent factors. Specifically, we build regression trees on users and items respectively with user-generated reviews, and associate a latent profile to each node on the trees to represent users and items. With the growth of regression tree, the latent factors are gradually refined under the regularization imposed by the tree structure. As a result, we are able to track the creation of latent profiles by looking into the path of each factor on regression trees, which thus serves as an explanation for the resulting recommendations. Extensive experiments on two large collections of Amazon and Yelp reviews demonstrate the advantage of our model over several competitive baseline algorithms. Besides, our extensive user study also confirms the practical value of explainable recommendations generated by our model. △ Less

Submitted 3 June, 2019; originally announced June 2019.

Comments: In proceedings of SIGIR'19

Are Anchor Points Really Indispensable in Label-Noise Learning?

Authors: Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, Masashi Sugiyama

Abstract: In label-noise learning, \textit{noise transition matrix}, denoting the probabilities that clean labels flip into noisy labels, plays a central role in building \textit{statistically consistent classifiers}. Existing theories have shown that the transition matrix can be learned by exploiting \textit{anchor points} (i.e., data points that belong to a specific class almost surely). However, when the… ▽ More In label-noise learning, \textit{noise transition matrix}, denoting the probabilities that clean labels flip into noisy labels, plays a central role in building \textit{statistically consistent classifiers}. Existing theories have shown that the transition matrix can be learned by exploiting \textit{anchor points} (i.e., data points that belong to a specific class almost surely). However, when there are no anchor points, the transition matrix will be poorly learned, and those current consistent classifiers will significantly degenerate. In this paper, without employing anchor points, we propose a \textit{transition-revision} ($T$-Revision) method to effectively learn transition matrices, leading to better classifiers. Specifically, to learn a transition matrix, we first initialize it by exploiting data points that are similar to anchor points, having high \textit{noisy class posterior probabilities}. Then, we modify the initialized matrix by adding a \textit{slack variable}, which can be learned and validated together with the classifier by using noisy data. Empirical results on benchmark-simulated and real-world label-noise datasets demonstrate that without using exact anchor points, the proposed method is superior to the state-of-the-art label-noise learning methods. △ Less

Submitted 16 December, 2019; v1 submitted 1 June, 2019; originally announced June 2019.

Comments: Accepted by NeurIPS 2019

EasiCS: the objective and fine-grained classification method of cervical spondylosis dysfunction

Authors: Nana Wang, Li Cui, Xi Huang, Yingcong Xiang, Jing Xiao, Yi Rao

Abstract: The precise diagnosis is of great significance in developing precise treatment plans to restore neck function and reduce the burden posed by the cervical spondylosis (CS). However, the current available neck function assessment method are subjective and coarse-grained. In this paper, based on the relationship among CS, cervical structure, cervical vertebra function, and surface electromyography (s… ▽ More The precise diagnosis is of great significance in developing precise treatment plans to restore neck function and reduce the burden posed by the cervical spondylosis (CS). However, the current available neck function assessment method are subjective and coarse-grained. In this paper, based on the relationship among CS, cervical structure, cervical vertebra function, and surface electromyography (sEMG), we seek to develop a clustering algorithms on the sEMG data set collected from the clinical environment and implement the division. We proposed and developed the framework EasiCS, which consists of dimension reduction, clustering algorithm EasiSOM, spectral clustering algorithm EasiSC. The EasiCS outperform the commonly used seven algorithms overall. △ Less

Submitted 15 May, 2019; originally announced May 2019.

Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks

Authors: Charbel Sakr, Naigang Wang, Chia-Yu Chen, Jungwook Choi, Ankur Agrawal, Naresh Shanbhag, Kailash Gopalakrishnan

Abstract: Efforts to reduce the numerical precision of computations in deep learning training have yielded systems that aggressively quantize weights and activations, yet employ wide high-precision accumulators for partial sums in inner-product operations to preserve the quality of convergence. The absence of any framework to analyze the precision requirements of partial sum accumulations results in conserv… ▽ More Efforts to reduce the numerical precision of computations in deep learning training have yielded systems that aggressively quantize weights and activations, yet employ wide high-precision accumulators for partial sums in inner-product operations to preserve the quality of convergence. The absence of any framework to analyze the precision requirements of partial sum accumulations results in conservative design choices. This imposes an upper-bound on the reduction of complexity of multiply-accumulate units. We present a statistical approach to analyze the impact of reduced accumulation precision on deep learning training. Observing that a bad choice for accumulation precision results in loss of information that manifests itself as a reduction in variance in an ensemble of partial sums, we derive a set of equations that relate this variance to the length of accumulation and the minimum number of bits needed for accumulation. We apply our analysis to three benchmark networks: CIFAR-10 ResNet 32, ImageNet ResNet 18 and ImageNet AlexNet. In each case, with accumulation precision set in accordance with our proposed equations, the networks successfully converge to the single precision floating-point baseline. We also show that reducing accumulation precision further degrades the quality of the trained network, proving that our equations produce tight bounds. Overall this analysis enables precise tailoring of computation hardware to the application, yielding area- and power-optimal systems. △ Less

Submitted 19 January, 2019; originally announced January 2019.

Comments: Published as a conference paper in ICLR 2019

Micro- and Macro-Level Churn Analysis of Large-Scale Mobile Games

Authors: Xi Liu, Muhe Xie, Xidao Wen, Rui Chen, Yong Ge, Nick Duffield, Na Wang

Abstract: As mobile devices become more and more popular, mobile gaming has emerged as a promising market with billion-dollar revenues. A variety of mobile game platforms and services have been developed around the world. A critical challenge for these platforms and services is to understand the churn behavior in mobile games, which usually involves churn at micro level (between an app and a specific user)… ▽ More As mobile devices become more and more popular, mobile gaming has emerged as a promising market with billion-dollar revenues. A variety of mobile game platforms and services have been developed around the world. A critical challenge for these platforms and services is to understand the churn behavior in mobile games, which usually involves churn at micro level (between an app and a specific user) and macro level (between an app and all its users). Accurate micro-level churn prediction and macro-level churn ranking will benefit many stakeholders such as game developers, advertisers, and platform operators. In this paper, we present the first large-scale churn analysis for mobile games that supports both micro-level churn prediction and macro-level churn ranking. For micro-level churn prediction, in view of the common limitations of the state-of-the-art methods built upon traditional machine learning models, we devise a novel semi-supervised and inductive embedding model that jointly learns the prediction function and the embedding function for user-app relationships. We model these two functions by deep neural networks with a unique edge embedding technique that is able to capture both contextual information and relationship dynamics. We also design a novel attributed random walk technique that takes into consideration both topological adjacency and attribute similarities. To address macro-level churn ranking, we propose to construct a relationship graph with estimated micro-level churn probabilities as edge weights and adapt link analysis algorithms on the graph. We devise a simple algorithm SimSum and adapt two more advanced algorithms PageRank and HITS. The performance of our solutions for the two-level churn analysis problems is evaluated on real-world data collected from the Samsung Game Launcher platform. △ Less

Submitted 14 January, 2019; originally announced January 2019.

Comments: arXiv admin note: substantial text overlap with arXiv:1808.06573

Training Deep Neural Networks with 8-bit Floating Point Numbers

Authors: Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Yu Chen, Kailash Gopalakrishnan

Abstract: The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, unlike inference, training with numbers represented with less than 16 bits has bee… ▽ More The state-of-the-art hardware platforms for training Deep Neural Networks (DNNs) are moving from traditional single precision (32-bit) computations towards 16 bits of precision -- in large part due to the high energy efficiency and smaller bit storage associated with using reduced-precision representations. However, unlike inference, training with numbers represented with less than 16 bits has been challenging due to the need to maintain fidelity of the gradient computations during back-propagation. Here we demonstrate, for the first time, the successful training of DNNs using 8-bit floating point numbers while fully maintaining the accuracy on a spectrum of Deep Learning models and datasets. In addition to reducing the data and computation precision to 8 bits, we also successfully reduce the arithmetic precision for additions (used in partial product accumulation and weight updates) from 32 bits to 16 bits through the introduction of a number of key ideas including chunk-based accumulation and floating point stochastic rounding. The use of these novel techniques lays the foundation for a new generation of hardware training platforms with the potential for 2-4x improved throughput over today's systems. △ Less

Submitted 19 December, 2018; originally announced December 2018.

Comments: NeurIPS 2018 (12 pages)

EasiCSDeep: A deep learning model for Cervical Spondylosis Identification using surface electromyography signal

Authors: Nana Wang, Li Cui, Xi Huang, Yingcong Xiang, Jing Xiao

Abstract: Cervical spondylosis (CS) is a common chronic disease that affects up to two-thirds of the population and poses a serious burden on individuals and society. The early identification has significant value in improving cure rate and reducing costs. However, the pathology is complex, and the mild symptoms increase the difficulty of the diagnosis, especially in the early stage. Besides, the time-consu… ▽ More Cervical spondylosis (CS) is a common chronic disease that affects up to two-thirds of the population and poses a serious burden on individuals and society. The early identification has significant value in improving cure rate and reducing costs. However, the pathology is complex, and the mild symptoms increase the difficulty of the diagnosis, especially in the early stage. Besides, the time-consuming and costliness of hospital medical service reduces the attention to the CS identification. Thus, a convenient, low-cost intelligent CS identification method is imperious demanded. In this paper, we present an intelligent method based on the deep learning to identify CS, using the surface electromyography (sEMG) signal. Faced with the complex, high dimensionality and weak usability of the sEMG signal, we proposed and developed a multi-channel EasiCSDeep algorithm based on the convolutional neural network, which consists of the feature extraction, spatial relationship representation and classification algorithm. To the best of our knowledge, this EasiCSDeep is the first effort to employ the deep learning and the sEMG data to identify CS. Compared with previous state-of-the-art algorithm, our algorithm achieves a significant improvement. △ Less

Submitted 12 December, 2018; originally announced December 2018.

Deep Density-based Image Clustering

Authors: Yazhou Ren, Ni Wang, Mingxia Li, Zenglin Xu

Abstract: Recently, deep clustering, which is able to perform feature learning that favors clustering tasks via deep neural networks, has achieved remarkable performance in image clustering applications. However, the existing deep clustering algorithms generally need the number of clusters in advance, which is usually unknown in real-world tasks. In addition, the initial cluster centers in the learned featu… ▽ More Recently, deep clustering, which is able to perform feature learning that favors clustering tasks via deep neural networks, has achieved remarkable performance in image clustering applications. However, the existing deep clustering algorithms generally need the number of clusters in advance, which is usually unknown in real-world tasks. In addition, the initial cluster centers in the learned feature space are generated by $k$-means. This only works well on spherical clusters and probably leads to unstable clustering results. In this paper, we propose a two-stage deep density-based image clustering (DDC) framework to address these issues. The first stage is to train a deep convolutional autoencoder (CAE) to extract low-dimensional feature representations from high-dimensional image data, and then apply t-SNE to further reduce the data to a 2-dimensional space favoring density-based clustering algorithms. The second stage is to apply the developed density-based clustering technique on the 2-dimensional embedded data to automatically recognize an appropriate number of clusters with arbitrary shapes. Concretely, a number of local clusters are generated to capture the local structures of clusters, and then are merged via their density relationship to form the final clustering result. Experiments demonstrate that the proposed DDC achieves comparable or even better clustering performance than state-of-the-art deep clustering methods, even though the number of clusters is not given. △ Less

Submitted 11 December, 2018; originally announced December 2018.

A Semi-Supervised and Inductive Embedding Model for Churn Prediction of Large-Scale Mobile Games

Authors: Xi Liu, Muhe Xie, Xidao Wen, Rui Chen, Yong Ge, Nick Duffield, Na Wang

Abstract: Mobile gaming has emerged as a promising market with billion-dollar revenues. A variety of mobile game platforms and services have been developed around the world. One critical challenge for these platforms and services is to understand user churn behavior in mobile games. Accurate churn prediction will benefit many stakeholders such as game developers, advertisers, and platform operators. In this… ▽ More Mobile gaming has emerged as a promising market with billion-dollar revenues. A variety of mobile game platforms and services have been developed around the world. One critical challenge for these platforms and services is to understand user churn behavior in mobile games. Accurate churn prediction will benefit many stakeholders such as game developers, advertisers, and platform operators. In this paper, we present the first large-scale churn prediction solution for mobile games. In view of the common limitations of the state-of-the-art methods built upon traditional machine learning models, we devise a novel semi-supervised and inductive embedding model that jointly learns the prediction function and the embedding function for user-app relationships. We model these two functions by deep neural networks with a unique edge embedding technique that is able to capture both contextual information and relationship dynamics. We also design a novel attributed random walk technique that takes into consideration both topological adjacency and attribute similarities. To evaluate the performance of our solution, we collect real-world data from the Samsung Game Launcher platform that includes tens of thousands of games and hundreds of millions of user-app interactions. The experimental results with this data demonstrate the superiority of our proposed model against existing state-of-the-art methods. △ Less

Submitted 10 October, 2018; v1 submitted 20 August, 2018; originally announced August 2018.

Comments: to appear in ICDM 2018

Code-division multiplexed resistive pulse sensor networks for spatio-temporal detection of particles in microfluidic devices

Authors: Ningquan Wang, Ruxiu Liu, Roozbeh Khodambashi, Norh Asmare, A. Fatih Sarioglu

Abstract: Spatial separation of suspended particles based on contrast in their physical or chemical properties forms the basis of various biological assays performed on lab-on-achip devices. To electronically acquire this information, we have recently introduced a microfluidic sensing platform, called Microfluidic CODES, which combines the resistive pulse sensing with the code division multiple access in mu… ▽ More Spatial separation of suspended particles based on contrast in their physical or chemical properties forms the basis of various biological assays performed on lab-on-achip devices. To electronically acquire this information, we have recently introduced a microfluidic sensing platform, called Microfluidic CODES, which combines the resistive pulse sensing with the code division multiple access in multiplexing a network of integrated electrical sensors. In this paper, we enhance the multiplexing capacity of the Microfluidic CODES by employing sensors that generate non-orthogonal code waveforms and a new decoding algorithm that combines machine learning techniques with minimum mean-squared error estimation. As a proof of principle, we fabricated a microfluidic device with a network of 10 code-multiplexed sensors and characterized it using cells suspended in phosphate buffer saline solution. △ Less

Submitted 9 August, 2018; originally announced August 2018.

Comments: 2017 IEEE 30th International Conference on Micro Electro Mechanical Systems (MEMS)

MSC Class: I.2.9

Empirical Evaluation of Rectified Activations in Convolutional Network

Authors: Bing Xu, Naiyan Wang, Tianqi Chen, Mu Li

Abstract: In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU). We evaluate these activation function on standard image classification task. Our experim… ▽ More In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU). We evaluate these activation function on standard image classification task. Our experiments suggest that incorporating a non-zero slope for negative part in rectified activation units could consistently improve the results. Thus our findings are negative on the common belief that sparsity is the key of good performance in ReLU. Moreover, on small scale dataset, using deterministic negative slope or learning it are both prone to overfitting. They are not as effective as using their randomized counterpart. By using RReLU, we achieved 75.68\% accuracy on CIFAR-100 test set without multiple test or ensemble. △ Less

Submitted 27 November, 2015; v1 submitted 4 May, 2015; originally announced May 2015.

A local approach to estimation in discrete loglinear models

Authors: Helene Massam, Nanwei Wang

Abstract: We consider two connected aspects of maximum likelihood estimation of the parameter for high-dimensional discrete graphical models: the existence of the maximum likelihood estimate (mle) and its computation. When the data is sparse, there are many zeros in the contingency table and the maximum likelihood estimate of the parameter may not exist. Fienberg and Rinaldo (2012) have shown that the mle… ▽ More We consider two connected aspects of maximum likelihood estimation of the parameter for high-dimensional discrete graphical models: the existence of the maximum likelihood estimate (mle) and its computation. When the data is sparse, there are many zeros in the contingency table and the maximum likelihood estimate of the parameter may not exist. Fienberg and Rinaldo (2012) have shown that the mle does not exists iff the data vector belongs to a face of the so-called marginal cone spanned by the rows of the design matrix of the model. Identifying these faces in high-dimension is challenging. In this paper, we take a local approach : we show that one such face, albeit possibly not the smallest one, can be identified by looking at a collection of marginal graphical models generated by induced subgraphs $G_i,i=1,\ldots,k$ of $G$. This is our first contribution. Our second contribution concerns the composite maximum likelihood estimate. When the dimension of the problem is large, estimating the parameters of a given graphical model through maximum likelihood is challenging, if not impossible. The traditional approach to this problem has been local with the use of composite likelihood based on local conditional likelihoods. A more recent development is to have the components of the composite likelihood be marginal likelihoods centred around each $v$. We first show that the estimates obtained by consensus through local conditional and marginal likelihoods are identical. We then study the asymptotic properties of the composite maximum likelihood estimate when both the dimension of the model and the sample size $N$ go to infinity. △ Less

Submitted 21 April, 2015; originally announced April 2015.

Comments: 36 pages, 1 figure and 3 tables

MSC Class: 62H17; 62M40

Collaborative Deep Learning for Recommender Systems

Authors: Hao Wang, Naiyan Wang, Dit-Yan Yeung

Abstract: Collaborative filtering (CF) is a successful approach commonly used by many recommender systems. Conventional CF-based methods use the ratings given to items by users as the sole source of information for learning to make recommendation. However, the ratings are often very sparse in many applications, causing CF-based methods to degrade significantly in their recommendation performance. To address… ▽ More Collaborative filtering (CF) is a successful approach commonly used by many recommender systems. Conventional CF-based methods use the ratings given to items by users as the sole source of information for learning to make recommendation. However, the ratings are often very sparse in many applications, causing CF-based methods to degrade significantly in their recommendation performance. To address this sparsity problem, auxiliary information such as item content information may be utilized. Collaborative topic regression (CTR) is an appealing recent method taking this approach which tightly couples the two components that learn from two different sources of information. Nevertheless, the latent representation learned by CTR may not be very effective when the auxiliary information is very sparse. To address this problem, we generalize recent advances in deep learning from i.i.d. input to non-i.i.d. (CF-based) input and propose in this paper a hierarchical Bayesian model called collaborative deep learning (CDL), which jointly performs deep representation learning for the content information and collaborative filtering for the ratings (feedback) matrix. Extensive experiments on three real-world datasets from different domains show that CDL can significantly advance the state of the art. △ Less

Submitted 18 June, 2015; v1 submitted 9 September, 2014; originally announced September 2014.

Convex Analysis of Mixtures for Separating Non-negative Well-grounded Sources

Authors: Yitan Zhu, Niya Wang, David J. Miller, Yue Wang

Abstract: Blind Source Separation (BSS) has proven to be a powerful tool for the analysis of composite patterns in engineering and science. We introduce Convex Analysis of Mixtures (CAM) for separating non-negative well-grounded sources, which learns the mixing matrix by identifying the lateral edges of the convex data scatter plot. We prove a sufficient and necessary condition for identifying the mixing ma… ▽ More Blind Source Separation (BSS) has proven to be a powerful tool for the analysis of composite patterns in engineering and science. We introduce Convex Analysis of Mixtures (CAM) for separating non-negative well-grounded sources, which learns the mixing matrix by identifying the lateral edges of the convex data scatter plot. We prove a sufficient and necessary condition for identifying the mixing matrix through edge detection, which also serves as the foundation for CAM to be applied not only to the exact-determined and over-determined cases, but also to the under-determined case. We show the optimality of the edge detection strategy, even for cases where source well-groundedness is not strictly satisfied. The CAM algorithm integrates plug-in noise filtering using sector-based clustering, an efficient geometric convex analysis scheme, and stability-based model order selection. We demonstrate the principle of CAM on simulated data and numerically mixed natural images. The superior performance of CAM against a panel of benchmark BSS techniques is demonstrated on numerically mixed gene expression data. We then apply CAM to dissect dynamic contrast-enhanced magnetic resonance imaging data taken from breast tumors and time-course microarray gene expression data derived from in-vivo muscle regeneration in mice, both producing biologically plausible decomposition results. △ Less

Submitted 10 December, 2015; v1 submitted 27 June, 2014; originally announced June 2014.

Comments: 15 pages, 9 figures, 2 tables

Gaussian-binary Restricted Boltzmann Machines on Modeling Natural Image Statistics

Authors: Nan Wang, Jan Melchior, Laurenz Wiskott

Abstract: We present a theoretical analysis of Gaussian-binary restricted Boltzmann machines (GRBMs) from the perspective of density models. The key aspect of this analysis is to show that GRBMs can be formulated as a constrained mixture of Gaussians, which gives a much better insight into the model's capabilities and limitations. We show that GRBMs are capable of learning meaningful features both in a two-… ▽ More We present a theoretical analysis of Gaussian-binary restricted Boltzmann machines (GRBMs) from the perspective of density models. The key aspect of this analysis is to show that GRBMs can be formulated as a constrained mixture of Gaussians, which gives a much better insight into the model's capabilities and limitations. We show that GRBMs are capable of learning meaningful features both in a two-dimensional blind source separation task and in modeling natural images. Further, we show that reported difficulties in training GRBMs are due to the failure of the training algorithm rather than the model itself. Based on our analysis we are able to propose several training recipes, which allowed successful and fast training in our experiments. Finally, we discuss the relationship of GRBMs to several modifications that have been proposed to improve the model. △ Less

Submitted 23 January, 2014; originally announced January 2014.

Comments: Current version is only an early manuscript and is subject to further change

Journal ref: PLoS ONE 12(2): e0171015 (2017)

A feasible roadmap for unsupervised deconvolution of two-source mixed gene expressions

Authors: Niya Wang, Eric P. Hoffman, Robert Clarke, Zhen Zhang, David M. Herrington, Ie-Ming Shih, Douglas A. Levine, Guoqiang Yu, Jianhua Xuan, Yue Wang

Abstract: Tissue heterogeneity is a major confounding factor in studying individual populations that cannot be resolved directly by global profiling. Experimental solutions to mitigate tissue heterogeneity are expensive, time consuming, inapplicable to existing data, and may alter the original gene expression patterns. Here we ask whether it is possible to deconvolute two-source mixed expressions (estimatin… ▽ More Tissue heterogeneity is a major confounding factor in studying individual populations that cannot be resolved directly by global profiling. Experimental solutions to mitigate tissue heterogeneity are expensive, time consuming, inapplicable to existing data, and may alter the original gene expression patterns. Here we ask whether it is possible to deconvolute two-source mixed expressions (estimating both proportions and cell-specific profiles) from two or more heterogeneous samples without requiring any prior knowledge. Supported by a well-grounded mathematical framework, we argue that both constituent proportions and cell-specific expressions can be estimated in a completely unsupervised mode when cell-specific marker genes exist, which do not have to be known a priori, for each of constituent cell types. We demonstrate the performance of unsupervised deconvolution on both simulation and real gene expression data, together with perspective discussions. △ Less

Submitted 25 October, 2013; originally announced October 2013.

Comments: 5 pages, 5 figures, 3 tables

Distributed parameter estimation of discrete hierarchical models via marginal likelihoods

Authors: Helene Massam, Nanwei Wang

Abstract: We consider discrete graphical models Markov with respect to a graph $G$ and propose two distributed marginal methods to estimate the maximum likelihood estimate of the canonical parameter of the model. Both methods are based on a relaxation of the marginal likelihood obtained by considering the density of the variables represented by a vertex $v$ of $G$ and a neighborhood. The two methods differ… ▽ More We consider discrete graphical models Markov with respect to a graph $G$ and propose two distributed marginal methods to estimate the maximum likelihood estimate of the canonical parameter of the model. Both methods are based on a relaxation of the marginal likelihood obtained by considering the density of the variables represented by a vertex $v$ of $G$ and a neighborhood. The two methods differ by the size of the neighborhood of $v$. We show that the estimates are consistent and that those obtained with the larger neighborhood have smaller asymptotic variance than the ones obtained through the smaller neighborhood. △ Less

Submitted 21 October, 2013; originally announced October 2013.

Comments: 21 pages, 7 figures

MSC Class: 62H17 (Primary); 62M40

Unsupervised deconvolution of dynamic imaging reveals intratumor vascular heterogeneity

Authors: Li Chen, Peter L. Choyke, Niya Wang, Robert Clarke, Zaver M. Bhujwalla, Elizabeth M. C. Hillman, Yue Wang

Abstract: Intratumor heterogeneity is often manifested by vascular compartments with distinct pharmacokinetics that cannot be resolved directly by in vivo dynamic imaging. We developed tissue-specific compartment modeling (TSCM), an unsupervised computational method of deconvolving dynamic imaging series from heterogeneous tumors that can improve vascular phenotyping in many biological contexts. Applying TS… ▽ More Intratumor heterogeneity is often manifested by vascular compartments with distinct pharmacokinetics that cannot be resolved directly by in vivo dynamic imaging. We developed tissue-specific compartment modeling (TSCM), an unsupervised computational method of deconvolving dynamic imaging series from heterogeneous tumors that can improve vascular phenotyping in many biological contexts. Applying TSCM to dynamic contrast-enhanced MRI of breast cancers revealed characteristic intratumor vascular heterogeneity and therapeutic responses that were otherwise undetectable. △ Less

Submitted 4 September, 2014; v1 submitted 14 June, 2013; originally announced June 2013.

Comments: Content: main manuscript, 31 pages

On The Degrees of Freedom of Reduced-rank Estimators in Multivariate Regression

Authors: Ashin Mukherjee, Kun Chen, Naisyin Wang, Ji Zhu

Abstract: In this paper we study the effective degrees of freedom of a general class of reduced rank estimators for multivariate regression in the framework of Stein's unbiased risk estimation (SURE). We derive a finite-sample exact unbiased estimator that admits a closed-form expression in terms of the singular values or thresholded singular values of the least squares solution and hence readily computable… ▽ More In this paper we study the effective degrees of freedom of a general class of reduced rank estimators for multivariate regression in the framework of Stein's unbiased risk estimation (SURE). We derive a finite-sample exact unbiased estimator that admits a closed-form expression in terms of the singular values or thresholded singular values of the least squares solution and hence readily computable. The results continue to hold in the high-dimensional scenario when both the predictor and response dimensions are allowed to be larger than the sample size. The derived analytical form facilitates the investigation of its theoretical properties and provides new insights into the empirical behaviors of the degrees of freedom. In particular, we examine the differences and connections between the proposed estimator and a commonly-used naive estimator, i.e., the number of free parameters. The use of the proposed estimator leads to efficient and accurate prediction risk estimation and model selection, as demonstrated by simulation studies and a data example. △ Less

Submitted 19 April, 2013; v1 submitted 8 October, 2012; originally announced October 2012.

Comments: 29 pages, 3 figures