-
On Universality of Non-Separable Approximate Message Passing Algorithms
Authors:
Max Lovig,
Tianhao Wang,
Zhou Fan
Abstract:
Mean-field characterizations of first-order iterative algorithms -- including Approximate Message Passing (AMP), stochastic and proximal gradient descent, and Langevin diffusions -- have enabled a precise understanding of learning dynamics in many statistical applications. For algorithms whose non-linearities have a coordinate-separable form, it is known that such characterizations enjoy a degree…
▽ More
Mean-field characterizations of first-order iterative algorithms -- including Approximate Message Passing (AMP), stochastic and proximal gradient descent, and Langevin diffusions -- have enabled a precise understanding of learning dynamics in many statistical applications. For algorithms whose non-linearities have a coordinate-separable form, it is known that such characterizations enjoy a degree of universality with respect to the underlying data distribution. However, mean-field characterizations of non-separable algorithm dynamics have largely remained restricted to i.i.d. Gaussian or rotationally-invariant data.
In this work, we initiate a study of universality for non-separable AMP algorithms. We identify a general condition for AMP with polynomial non-linearities, in terms of a Bounded Composition Property (BCP) for their representing tensors, to admit a state evolution that holds universally for matrices with non-Gaussian entries. We then formalize a condition of BCP-approximability for Lipschitz AMP algorithms to enjoy a similar universal guarantee. We demonstrate that many common classes of non-separable non-linearities are BCP-approximable, including local denoisers, spectral denoisers for generic signals, and compositions of separable functions with generic linear maps, implying the universality of state evolution for AMP algorithms employing these non-linearities.
△ Less
Submitted 28 June, 2025;
originally announced June 2025.
-
Almost-Optimal Local-Search Methods for Sparse Tensor PCA
Authors:
Max Lovig,
Conor Sheehan,
Konstantinos Tsirkas,
Ilias Zadik
Abstract:
Local-search methods are widely employed in statistical applications, yet interestingly, their theoretical foundations remain rather underexplored, compared to other classes of estimators such as low-degree polynomials and spectral methods. Of note, among the few existing results recent studies have revealed a significant "local-computational" gap in the context of a well-studied sparse tensor pri…
▽ More
Local-search methods are widely employed in statistical applications, yet interestingly, their theoretical foundations remain rather underexplored, compared to other classes of estimators such as low-degree polynomials and spectral methods. Of note, among the few existing results recent studies have revealed a significant "local-computational" gap in the context of a well-studied sparse tensor principal component analysis (PCA), where a broad class of local Markov chain methods exhibits a notable underperformance relative to other polynomial-time algorithms. In this work, we propose a series of local-search methods that provably "close" this gap to the best known polynomial-time procedures in multiple regimes of the model, including and going beyond the previously studied regimes in which the broad family of local Markov chain methods underperforms. Our framework includes: (1) standard greedy and randomized greedy algorithms applied to the (regularized) posterior of the model; and (2) novel random-threshold variants, in which the randomized greedy algorithm accepts a proposed transition if and only if the corresponding change in the Hamiltonian exceeds a random Gaussian threshold-rather that if and only if it is positive, as is customary. The introduction of the random thresholds enables a tight mathematical analysis of the randomized greedy algorithm's trajectory by crucially breaking the dependencies between the iterations, and could be of independent interest to the community.
△ Less
Submitted 11 June, 2025;
originally announced June 2025.
-
On The MCMC Performance In Bernoulli Group Testing And The Random Max Set-Cover Problem
Authors:
Maxwell Lovig,
Ilias Zadik
Abstract:
The group testing problem is a canonical inference task where one seeks to identify $k$ infected individuals out of a population of $n$ people, based on the outcomes of $m$ group tests. Of particular interest is the case of Bernoulli group testing (BGT), where each individual participates in each test independently and with a fixed probability. BGT is known to be an "information-theoretically" opt…
▽ More
The group testing problem is a canonical inference task where one seeks to identify $k$ infected individuals out of a population of $n$ people, based on the outcomes of $m$ group tests. Of particular interest is the case of Bernoulli group testing (BGT), where each individual participates in each test independently and with a fixed probability. BGT is known to be an "information-theoretically" optimal design, as there exists a decoder that can identify with high probability as $n$ grows the infected individuals using $m^*=\log_2 \binom{n}{k}$ BGT tests, which is the minimum required number of tests among \emph{all} group testing designs.
An important open question in the field is if a polynomial-time decoder exists for BGT which succeeds also with $m^*$ samples. In a recent paper (Iliopoulos, Zadik COLT '21) some evidence was presented (but no proof) that a simple low-temperature MCMC method could succeed. The evidence was based on a first-moment (or "annealed") analysis of the landscape, as well as simulations that show the MCMC success for $n \approx 1000s$.
In this work, we prove that, despite the intriguing success in simulations for small $n$, the class of MCMC methods proposed in previous work for BGT with $m^*$ samples takes super-polynomial-in-$n$ time to identify the infected individuals, when $k=n^α$ for $α\in (0,1)$ small enough. Towards obtaining our results, we establish the tight max-satisfiability thresholds of the random $k$-set cover problem, a result of potentially independent interest in the study of random constraint satisfaction problems.
△ Less
Submitted 16 October, 2024; v1 submitted 11 October, 2024;
originally announced October 2024.
