-
Weak Poincaré Inequalities, Simulated Annealing, and Sampling from Spherical Spin Glasses
Authors:
Brice Huang,
Sidhanth Mohanty,
Amit Rajaraman,
David X. Wu
Abstract:
There has been a recent surge of powerful tools to show rapid mixing of Markov chains, via functional inequalities such as Poincaré inequalities. In many situations, Markov chains fail to mix rapidly from a worst-case initialization, yet are expected to approximately sample from a random initialization. For example, this occurs if the target distribution has metastable states, small clusters accou…
▽ More
There has been a recent surge of powerful tools to show rapid mixing of Markov chains, via functional inequalities such as Poincaré inequalities. In many situations, Markov chains fail to mix rapidly from a worst-case initialization, yet are expected to approximately sample from a random initialization. For example, this occurs if the target distribution has metastable states, small clusters accounting for a vanishing fraction of the mass that are essentially disconnected from the bulk of the measure. Under such conditions, a Poincaré inequality cannot hold, necessitating new tools to prove sampling guarantees.
We develop a framework to analyze simulated annealing, based on establishing so-called weak Poincaré inequalities. These inequalities imply mixing from a suitably warm start, and simulated annealing provides a way to chain such warm starts together into a sampling algorithm. We further identify a local-to-global principle to prove weak Poincaré inequalities, mirroring the spectral independence and localization schemes frameworks for analyzing mixing times of Markov chains.
As our main application, we prove that simulated annealing samples from the Gibbs measure of a spherical spin glass for inverse temperatures up to a natural threshold, matching recent algorithms based on algorithmic stochastic localization. This provides the first Markov chain sampling guarantee that holds beyond the uniqueness threshold for spherical spin glasses, where mixing from a worst-case initialization is provably slow due to the presence of metastable states. As an ingredient in our proof, we prove bounds on the operator norm of the covariance matrix of spherical spin glasses in the full replica-symmetric regime.
Additionally, we resolve a question related to sampling using data-based initializations.
△ Less
Submitted 22 November, 2024; v1 submitted 13 November, 2024;
originally announced November 2024.
-
Locally Stationary Distributions: A Framework for Analyzing Slow-Mixing Markov Chains
Authors:
Kuikui Liu,
Sidhanth Mohanty,
Prasad Raghavendra,
Amit Rajaraman,
David X. Wu
Abstract:
Many natural Markov chains fail to mix to their stationary distribution in polynomially many steps. Often, this slow mixing is inevitable since it is computationally intractable to sample from their stationary measure.
Nevertheless, Markov chains can be shown to always converge quickly to measures that are locally stationary, i.e., measures that don't change over a small number of steps. These l…
▽ More
Many natural Markov chains fail to mix to their stationary distribution in polynomially many steps. Often, this slow mixing is inevitable since it is computationally intractable to sample from their stationary measure.
Nevertheless, Markov chains can be shown to always converge quickly to measures that are locally stationary, i.e., measures that don't change over a small number of steps. These locally stationary measures are analogous to local minima in continuous optimization, while stationary measures correspond to global minima.
While locally stationary measures can be statistically far from stationary measures, do they enjoy provable theoretical guarantees that have algorithmic implications? We study this question in this work and demonstrate three algorithmic applications of locally stationary measures:
1. We show that Glauber dynamics on the hardcore model can be used to find independent sets of size $Ω\left(\frac{\log d}{d} \cdot n\right)$ in triangle-free graphs of degree at most $d$.
2. Let $W$ be a symmetric real matrix with bounded spectral diameter and $v$ be a unit vector. Given the matrix $M = λvv^\top + W$ with a planted rank-one spike along vector $v$, for sufficiently large constant $λ$, Glauber dynamics on the Ising model defined by $M$ samples vectors $x \in \{\pm 1\}^n$ that have constant correlation with the vector $v$.
3. Let $M = A_{\mathbf{G}} - \frac{d}{n}\mathbf{1}\mathbf{1}^\top$ be a centered version of the adjacency matrix where the graph $\mathbf{G}$ is drawn from a sparse 2-community stochastic block model. We show that for sufficiently large constant signal-to-noise ratio, Glauber dynamics on the Ising model defined by $M$ samples vectors $x \in \{\pm 1\}^n$ that have constant correlation with the hidden community vector $\mathbfσ$.
△ Less
Submitted 8 April, 2025; v1 submitted 31 May, 2024;
originally announced May 2024.
-
Fast Mixing in Sparse Random Ising Models
Authors:
Kuikui Liu,
Sidhanth Mohanty,
Amit Rajaraman,
David X. Wu
Abstract:
Motivated by the community detection problem in Bayesian inference, as well as the recent explosion of interest in spin glasses from statistical physics, we study the classical Glauber dynamics for sampling from Ising models with sparse random interactions. It is now well-known that when the interaction matrix has spectral diameter less than $1$, Glauber dynamics mixes in $O(n\log n)$ steps. Unfor…
▽ More
Motivated by the community detection problem in Bayesian inference, as well as the recent explosion of interest in spin glasses from statistical physics, we study the classical Glauber dynamics for sampling from Ising models with sparse random interactions. It is now well-known that when the interaction matrix has spectral diameter less than $1$, Glauber dynamics mixes in $O(n\log n)$ steps. Unfortunately, such criteria fail dramatically for interactions supported on arguably the most well-studied sparse random graph: the Erdős--Rényi random graph $G(n,d/n)$, due to the presence of almost linearly many outlier eigenvalues of unbounded magnitude.
We prove that for the \emph{Viana--Bray spin glass}, where the interactions are supported on $G(n,d/n)$ and randomly assigned $\pmβ$, Glauber dynamics mixes in $n^{1+o(1)}$ time with high probability as long as $β\le O(1/\sqrt{d})$, independent of $n$. We further extend our results to random graphs drawn according to the $2$-community stochastic block model, as well as when the interactions are given by a "centered" version of the adjacency matrix. The latter setting is particularly relevant for the inference problem in community detection. Indeed, we use this to show that Glauber dynamics succeeds at recovering communities in the stochastic block model in a companion paper [LMR+24].
The primary technical ingredient in our proof is showing that with high probability, a sparse random graph can be decomposed into two parts -- a \emph{bulk} which behaves like a graph with bounded maximum degree and a well-behaved spectrum, and a \emph{near-forest} with favorable pseudorandom properties. We then use this decomposition to design a localization procedure that interpolates to simpler Ising models supported only on the near-forest, and then execute a pathwise analysis to establish a modified log-Sobolev inequality.
△ Less
Submitted 5 August, 2024; v1 submitted 10 May, 2024;
originally announced May 2024.
-
Robust recovery for stochastic block models, simplified and generalized
Authors:
Sidhanth Mohanty,
Prasad Raghavendra,
David X. Wu
Abstract:
We study the problem of $\textit{robust community recovery}$: efficiently recovering communities in sparse stochastic block models in the presence of adversarial corruptions. In the absence of adversarial corruptions, there are efficient algorithms when the $\textit{signal-to-noise ratio}$ exceeds the $\textit{Kesten--Stigum (KS) threshold}$, widely believed to be the computational threshold for t…
▽ More
We study the problem of $\textit{robust community recovery}$: efficiently recovering communities in sparse stochastic block models in the presence of adversarial corruptions. In the absence of adversarial corruptions, there are efficient algorithms when the $\textit{signal-to-noise ratio}$ exceeds the $\textit{Kesten--Stigum (KS) threshold}$, widely believed to be the computational threshold for this problem. The question we study is: does the computational threshold for robust community recovery also lie at the KS threshold? We answer this question affirmatively, providing an algorithm for robust community recovery for arbitrary stochastic block models on any constant number of communities, generalizing the work of Ding, d'Orsi, Nasser & Steurer on an efficient algorithm above the KS threshold in the case of $2$-community block models.
There are three main ingredients to our work:
(i) The Bethe Hessian of the graph is defined as $H_G(t) \triangleq (D_G-I)t^2 - A_Gt + I$ where $D_G$ is the diagonal matrix of degrees and $A_G$ is the adjacency matrix. Empirical work suggested that the Bethe Hessian for the stochastic block model has outlier eigenvectors corresponding to the communities right above the Kesten-Stigum threshold. We formally confirm the existence of outlier eigenvalues for the Bethe Hessian, by explicitly constructing outlier eigenvectors from the community vectors.
(ii) We develop an algorithm for a variant of robust PCA on sparse matrices. Specifically, an algorithm to partially recover top eigenspaces from adversarially corrupted sparse matrices under mild delocalization constraints.
(iii) A rounding algorithm to turn vector assignments of vertices into a community assignment, inspired by the algorithm of Charikar \& Wirth \cite{CW04} for $2$XOR.
△ Less
Submitted 21 February, 2024;
originally announced February 2024.
-
On the Training Instability of Shuffling SGD with Batch Normalization
Authors:
David X. Wu,
Chulhee Yun,
Suvrit Sra
Abstract:
We uncover how SGD interacts with batch normalization and can exhibit undesirable training dynamics such as divergence. More precisely, we study how Single Shuffle (SS) and Random Reshuffle (RR) -- two widely used variants of SGD -- interact surprisingly differently in the presence of batch normalization: RR leads to much more stable evolution of training loss than SS. As a concrete example, for r…
▽ More
We uncover how SGD interacts with batch normalization and can exhibit undesirable training dynamics such as divergence. More precisely, we study how Single Shuffle (SS) and Random Reshuffle (RR) -- two widely used variants of SGD -- interact surprisingly differently in the presence of batch normalization: RR leads to much more stable evolution of training loss than SS. As a concrete example, for regression using a linear network with batch normalization, we prove that SS and RR converge to distinct global optima that are "distorted" away from gradient descent. Thereafter, for classification we characterize conditions under which training divergence for SS and RR can, and cannot occur. We present explicit constructions to show how SS leads to distorted optima in regression and divergence for classification, whereas RR avoids both distortion and divergence. We validate our results by confirming them empirically in realistic settings, and conclude that the separation between SS and RR used with batch normalization is relevant in practice.
△ Less
Submitted 14 August, 2023; v1 submitted 23 February, 2023;
originally announced February 2023.
