-
A Generalized Tangent Approximation Framework for Strongly Super-Gaussian Likelihoods
Authors:
Somjit Roy,
Pritam Dey,
Debdeep Pati,
Bani K. Mallick
Abstract:
Tangent approximation form a popular class of variational inference (VI) techniques for Bayesian analysis in intractable non-conjugate models. It is based on the principle of convex duality to construct a minorant of the marginal likelihood, making the problem tractable. Despite its extensive applications, a general methodology for tangent approximation encompassing a large class of likelihoods be…
▽ More
Tangent approximation form a popular class of variational inference (VI) techniques for Bayesian analysis in intractable non-conjugate models. It is based on the principle of convex duality to construct a minorant of the marginal likelihood, making the problem tractable. Despite its extensive applications, a general methodology for tangent approximation encompassing a large class of likelihoods beyond logit models with provable optimality guarantees is still elusive. In this article, we propose a general Tangent Approximation based Variational InferencE (TAVIE) framework for strongly super-Gaussian (SSG) likelihood functions which includes a broad class of flexible probability models. Specifically, TAVIE obtains a quadratic lower bound of the corresponding log-likelihood, thus inducing conjugacy with Gaussian priors over the model parameters. Under mild assumptions on the data-generating process, we demonstrate the optimality of our proposed methodology in the fractional likelihood setup. Furthermore, we illustrate the empirical performance of TAVIE through extensive simulations and an application on the U.S. 2000 Census real data.
△ Less
Submitted 7 April, 2025;
originally announced April 2025.
-
Generalized Regret Analysis of Thompson Sampling using Fractional Posteriors
Authors:
Prateek Jaiswal,
Debdeep Pati,
Anirban Bhattacharya,
Bani K. Mallick
Abstract:
Thompson sampling (TS) is one of the most popular and earliest algorithms to solve stochastic multi-armed bandit problems. We consider a variant of TS, named $α$-TS, where we use a fractional or $α$-posterior ($α\in(0,1)$) instead of the standard posterior distribution. To compute an $α$-posterior, the likelihood in the definition of the standard posterior is tempered with a factor $α$. For $α$-TS…
▽ More
Thompson sampling (TS) is one of the most popular and earliest algorithms to solve stochastic multi-armed bandit problems. We consider a variant of TS, named $α$-TS, where we use a fractional or $α$-posterior ($α\in(0,1)$) instead of the standard posterior distribution. To compute an $α$-posterior, the likelihood in the definition of the standard posterior is tempered with a factor $α$. For $α$-TS we obtain both instance-dependent $\mathcal{O}\left(\sum_{k \neq i^*} Δ_k\left(\frac{\log(T)}{C(α)Δ_k^2} + \frac{1}{2} \right)\right)$ and instance-independent $\mathcal{O}(\sqrt{KT\log K})$ frequentist regret bounds under very mild conditions on the prior and reward distributions, where $Δ_k$ is the gap between the true mean rewards of the $k^{th}$ and the best arms, and $C(α)$ is a known constant. Both the sub-Gaussian and exponential family models satisfy our general conditions on the reward distribution. Our conditions on the prior distribution just require its density to be positive, continuous, and bounded. We also establish another instance-dependent regret upper bound that matches (up to constants) to that of improved UCB [Auer and Ortner, 2010]. Our regret analysis carefully combines recent theoretical developments in the non-asymptotic concentration analysis and Bernstein-von Mises type results for the $α$-posterior distribution. Moreover, our analysis does not require additional structural properties such as closed-form posteriors or conjugate priors.
△ Less
Submitted 12 September, 2023;
originally announced September 2023.
-
Structured Optimal Variational Inference for Dynamic Latent Space Models
Authors:
Peng Zhao,
Anirban Bhattacharya,
Debdeep Pati,
Bani K. Mallick
Abstract:
We consider a latent space model for dynamic networks, where our objective is to estimate the pairwise inner products plus the intercept of the latent positions. To balance posterior inference and computational scalability, we consider a structured mean-field variational inference framework, where the time-dependent properties of the dynamic networks are exploited to facilitate computation and inf…
▽ More
We consider a latent space model for dynamic networks, where our objective is to estimate the pairwise inner products plus the intercept of the latent positions. To balance posterior inference and computational scalability, we consider a structured mean-field variational inference framework, where the time-dependent properties of the dynamic networks are exploited to facilitate computation and inference. Additionally, an easy-to-implement block coordinate ascent algorithm is developed with message-passing type updates in each block, whereas the complexity per iteration is linear with the number of nodes and time points. To certify the optimality, we demonstrate that the variational risk of the proposed variational inference approach attains the minimax optimal rate with only a logarithm factor under certain conditions. To this end, we first derive the minimax lower bound, which might be of independent interest. In addition, we show that the posterior under commonly adopted Gaussian random walk priors can achieve the minimax lower bound with only a logarithm factor. To the best of our knowledge, this is the first such a throughout theoretical analysis of Bayesian dynamic latent space models. Simulations and real data analysis demonstrate the efficacy of our methodology and the efficiency of our algorithm.
△ Less
Submitted 15 October, 2024; v1 submitted 29 September, 2022;
originally announced September 2022.
-
Bayesian Variable Selection in Multivariate Nonlinear Regression with Graph Structures
Authors:
Yabo Niu,
Nilabja Guha,
Debkumar De,
Anindya Bhadra,
Veerabhadran Baladandayuthapani,
Bani K. Mallick
Abstract:
Gaussian graphical models (GGMs) are well-established tools for probabilistic exploration of dependence structures using precision matrices. We develop a Bayesian method to incorporate covariate information in this GGMs setup in a nonlinear seemingly unrelated regression framework. We propose a joint predictor and graph selection model and develop an efficient collapsed Gibbs sampler algorithm to…
▽ More
Gaussian graphical models (GGMs) are well-established tools for probabilistic exploration of dependence structures using precision matrices. We develop a Bayesian method to incorporate covariate information in this GGMs setup in a nonlinear seemingly unrelated regression framework. We propose a joint predictor and graph selection model and develop an efficient collapsed Gibbs sampler algorithm to search the joint model space. Furthermore, we investigate its theoretical variable selection properties. We demonstrate our method on a variety of simulated data, concluding with a real data set from the TCPA project.
△ Less
Submitted 30 July, 2021; v1 submitted 27 October, 2020;
originally announced October 2020.
-
Tail-adaptive Bayesian shrinkage
Authors:
Se Yoon Lee,
Peng Zhao,
Debdeep Pati,
Bani K. Mallick
Abstract:
Robust Bayesian methods for high-dimensional regression problems under diverse sparse regimes are studied. Traditional shrinkage priors are primarily designed to detect a handful of signals from tens of thousands of predictors in the so-called ultra-sparsity domain. However, they may not perform desirably when the degree of sparsity is moderate. In this paper, we propose a robust sparse estimation…
▽ More
Robust Bayesian methods for high-dimensional regression problems under diverse sparse regimes are studied. Traditional shrinkage priors are primarily designed to detect a handful of signals from tens of thousands of predictors in the so-called ultra-sparsity domain. However, they may not perform desirably when the degree of sparsity is moderate. In this paper, we propose a robust sparse estimation method under diverse sparsity regimes, which has a tail-adaptive shrinkage property. In this property, the tail-heaviness of the prior adjusts adaptively, becoming larger or smaller as the sparsity level increases or decreases, respectively, to accommodate more or fewer signals, a posteriori. We propose a global-local-tail (GLT) Gaussian mixture distribution that ensures this property. We examine the role of the tail-index of the prior in relation to the underlying sparsity level and demonstrate that the GLT posterior contracts at the minimax optimal rate for sparse normal mean models. We apply both the GLT prior and the Horseshoe prior to a real data problem and simulation examples. Our findings indicate that the varying tail rule based on the GLT prior offers advantages over a fixed tail rule based on the Horseshoe prior in diverse sparsity regimes.
△ Less
Submitted 24 October, 2024; v1 submitted 4 July, 2020;
originally announced July 2020.
-
Bayesian sparse multiple regression for simultaneous rank reduction and variable selection
Authors:
Antik Chakraborty,
Anirban Bhattacharya,
Bani K. Mallick
Abstract:
We develop a Bayesian methodology aimed at simultaneously estimating low-rank and row-sparse matrices in a high-dimensional multiple-response linear regression model. We consider a carefully devised shrinkage prior on the matrix of regression coefficients which obviates the need to specify a prior on the rank, and shrinks the regression matrix towards low-rank and row-sparse structures. We provide…
▽ More
We develop a Bayesian methodology aimed at simultaneously estimating low-rank and row-sparse matrices in a high-dimensional multiple-response linear regression model. We consider a carefully devised shrinkage prior on the matrix of regression coefficients which obviates the need to specify a prior on the rank, and shrinks the regression matrix towards low-rank and row-sparse structures. We provide theoretical support to the proposed methodology by proving minimax optimality of the posterior mean under the prediction risk in ultra-high dimensional settings where the number of predictors can grow sub-exponentially relative to the sample size. A one-step post-processing scheme induced by group lasso penalties on the rows of the estimated coefficient matrix is proposed for variable selection, with default choices of tuning parameters. We additionally provide an estimate of the rank using a novel optimization function achieving dimension reduction in the covariate space. We exhibit the performance of the proposed methodology in an extensive simulation study and a real data example.
△ Less
Submitted 8 April, 2019; v1 submitted 2 December, 2016;
originally announced December 2016.
-
Adaptive Posterior Convergence Rates in Bayesian Density Deconvolution with Supersmooth Errors
Authors:
Abhra Sarkar,
Debdeep Pati,
Bani K. Mallick,
Raymond J. Carroll
Abstract:
Bayesian density deconvolution using nonparametric prior distributions is a useful alternative to the frequentist kernel based deconvolution estimators due to its potentially wide range of applicability, straightforward uncertainty quantification and generalizability to more sophisticated models. This article is the first substantive effort to theoretically quantify the behavior of the posterior i…
▽ More
Bayesian density deconvolution using nonparametric prior distributions is a useful alternative to the frequentist kernel based deconvolution estimators due to its potentially wide range of applicability, straightforward uncertainty quantification and generalizability to more sophisticated models. This article is the first substantive effort to theoretically quantify the behavior of the posterior in this recent line of research. In particular, assuming a known supersmooth error density, a Dirichlet process mixture of Normals on the true density leads to a posterior convergence rate same as the minimax rate $(\log n)^{-η/β}$ adaptively over the smoothness $η$ of an appropriate Hölder space of densities, where $β$ is the degree of smoothness of the error distribution. Our main contribution is achieving adaptive minimax rates with respect to the $L_p$ norm for $2 \leq p \leq \infty$ under mild regularity conditions on the true density. En route, we develop tight concentration bounds for a class of kernel based deconvolution estimators which might be of independent interest.
△ Less
Submitted 9 September, 2013; v1 submitted 25 August, 2013;
originally announced August 2013.
