-
A greedy anytime algorithm for sparse PCA
Authors:
Guy Holtzman,
Adam Soffer,
Dan Vilenchik
Abstract:
The taxing computational effort that is involved in solving some high-dimensional statistical problems, in particular problems involving non-convex optimization, has popularized the development and analysis of algorithms that run efficiently (polynomial-time) but with no general guarantee on statistical consistency. In light of the ever-increasing compute power and decreasing costs, a more useful…
▽ More
The taxing computational effort that is involved in solving some high-dimensional statistical problems, in particular problems involving non-convex optimization, has popularized the development and analysis of algorithms that run efficiently (polynomial-time) but with no general guarantee on statistical consistency. In light of the ever-increasing compute power and decreasing costs, a more useful characterization of algorithms is by their ability to calibrate the invested computational effort with various characteristics of the input at hand and with the available computational resources. For example, design an algorithm that always guarantees statistical consistency of its output by increasing the running time as the SNR weakens. We propose a new greedy algorithm for the $\ell_0$-sparse PCA problem which supports the calibration principle. We provide both a rigorous analysis of our algorithm in the spiked covariance model, as well as simulation results and comparison with other existing methods. Our findings show that our algorithm recovers the spike in SNR regimes where all polynomial-time algorithms fail while running in a reasonable parallel-time on a cluster.
△ Less
Submitted 12 February, 2020; v1 submitted 15 October, 2019;
originally announced October 2019.
-
The condensation phase transition in random graph coloring
Authors:
Victor Bapst,
Amin Coja-Oghlan,
Samuel Hetterich,
Felicia Rassmann,
Dan Vilenchik
Abstract:
Based on a non-rigorous formalism called the "cavity method", physicists have put forward intriguing predictions on phase transitions in discrete structures. One of the most remarkable ones is that in problems such as random $k$-SAT or random graph $k$-coloring, very shortly before the threshold for the existence of solutions there occurs another phase transition called "condensation" [Krzakala et…
▽ More
Based on a non-rigorous formalism called the "cavity method", physicists have put forward intriguing predictions on phase transitions in discrete structures. One of the most remarkable ones is that in problems such as random $k$-SAT or random graph $k$-coloring, very shortly before the threshold for the existence of solutions there occurs another phase transition called "condensation" [Krzakala et al., PNAS 2007]. The existence of this phase transition appears to be intimately related to the difficulty of proving precise results on, e.g., the $k$-colorability threshold as well as to the performance of message passing algorithms. In random graph $k$-coloring, there is a precise conjecture as to the location of the condensation phase transition in terms of a distributional fixed point problem. In this paper we prove this conjecture for $k$ exceeding a certain constant $k_0$.
△ Less
Submitted 19 April, 2014;
originally announced April 2014.
-
Do semidefinite relaxations solve sparse PCA up to the information limit?
Authors:
Robert Krauthgamer,
Boaz Nadler,
Dan Vilenchik
Abstract:
Estimating the leading principal components of data, assuming they are sparse, is a central task in modern high-dimensional statistics. Many algorithms were developed for this sparse PCA problem, from simple diagonal thresholding to sophisticated semidefinite programming (SDP) methods. A key theoretical question is under what conditions can such algorithms recover the sparse principal components?…
▽ More
Estimating the leading principal components of data, assuming they are sparse, is a central task in modern high-dimensional statistics. Many algorithms were developed for this sparse PCA problem, from simple diagonal thresholding to sophisticated semidefinite programming (SDP) methods. A key theoretical question is under what conditions can such algorithms recover the sparse principal components? We study this question for a single-spike model with an $\ell_0$-sparse eigenvector, in the asymptotic regime as dimension $p$ and sample size $n$ both tend to infinity. Amini and Wainwright [Ann. Statist. 37 (2009) 2877-2921] proved that for sparsity levels $k\geqΩ(n/\log p)$, no algorithm, efficient or not, can reliably recover the sparse eigenvector. In contrast, for $k\leq O(\sqrt{n/\log p})$, diagonal thresholding is consistent. It was further conjectured that an SDP approach may close this gap between computational and information limits. We prove that when $k\geqΩ(\sqrt{n})$, the proposed SDP approach, at least in its standard usage, cannot recover the sparse spike. In fact, we conjecture that in the single-spike model, no computationally-efficient algorithm can recover a spike of $\ell_0$-sparsity $k\geqΩ(\sqrt{n})$. Finally, we present empirical results suggesting that up to sparsity levels $k=O(\sqrt{n})$, recovery is possible by a simple covariance thresholding algorithm.
△ Less
Submitted 3 June, 2015; v1 submitted 16 June, 2013;
originally announced June 2013.
-
Chasing the k-colorability threshold
Authors:
Amin Coja-Oghlan,
Dan Vilenchik
Abstract:
Over the past decade, physicists have developed deep but non-rigorous techniques for studying phase transitions in discrete structures. Recently, their ideas have been harnessed to obtain improved rigorous results on the phase transitions in binary problems such as random $k$-SAT or $k$-NAESAT (e.g., Coja-Oghlan and Panagiotou: STOC 2013). However, these rigorous arguments, typically centered arou…
▽ More
Over the past decade, physicists have developed deep but non-rigorous techniques for studying phase transitions in discrete structures. Recently, their ideas have been harnessed to obtain improved rigorous results on the phase transitions in binary problems such as random $k$-SAT or $k$-NAESAT (e.g., Coja-Oghlan and Panagiotou: STOC 2013). However, these rigorous arguments, typically centered around the second moment method, do not extend easily to problems where there are more than two possible values per variable. The single most intensely studied example of such a problem is random graph $k$-coloring. Here we develop a novel approach to the second moment method in this problem. This new method, inspired by physics conjectures on the geometry of the set of $k$-colorings, allows us to establish a substantially improved lower bound on the $k$-colorability threshold. The new lower bound is within an additive $2\ln 2+o_k(1)\approx 1.39$ of a simple first-moment upper bound and within $2\ln 2-1+o_k(1)\approx 0.39$ of the physics conjecture. By comparison, the best previous lower bound left a gap of about $2+\ln k$, unbounded in terms of the number of colors [Achlioptas, Naor: STOC 2004].
△ Less
Submitted 29 April, 2014; v1 submitted 3 April, 2013;
originally announced April 2013.
-
Edge distribution in generalized graph products
Authors:
Michael Langberg,
Dan Vilenchik
Abstract:
Given a graph $G=(V,E)$, an integer $k$, and a function $f_G:V^k \times V^k \to {0,1}$, the $k^{th}$ graph product of $G$ w.r.t $f_G$ is the graph with vertex set $V^k$, and an edge between two vertices $x=(x_1,...,x_k)$ and $y=(y_1,...,y_k)$ iff $f_G(x,y)=1$. Graph products are a basic combinatorial object, widely studied and used in different areas such as hardness of approximation, information…
▽ More
Given a graph $G=(V,E)$, an integer $k$, and a function $f_G:V^k \times V^k \to {0,1}$, the $k^{th}$ graph product of $G$ w.r.t $f_G$ is the graph with vertex set $V^k$, and an edge between two vertices $x=(x_1,...,x_k)$ and $y=(y_1,...,y_k)$ iff $f_G(x,y)=1$. Graph products are a basic combinatorial object, widely studied and used in different areas such as hardness of approximation, information theory, etc. We study graph products for functions $f_G$ of the form $f_G(x,y)=1$ iff there are at least $t$ indices $i \in [k]$ s.t. $(x_i,y_i)\in E$, where $t \in [k]$ is a fixed parameter in $f_G$. This framework generalizes the well-known graph tensor-product (obtained for $t=k$) and the graph or-product (obtained for $t=1$). The property that interests us is the edge distribution in such graphs. We show that if $G$ has a spectral gap, then the number of edges connecting "large-enough" sets in $G^k$ is "well-behaved", namely, it is close to the expected value, had the sets been random. We extend our results to bi-partite graph products as well. For a bi-partite graph $G=(X,Y,E)$, the $k^{th}$ bi-partite graph product of $G$ w.r.t $f_G$ is the bi-partite graph with vertex sets $X^k$ and $Y^k$ and edges between $x \in X^k$ and $y \in Y^k$ iff $f_G(x,y)=1$. Finally, for both types of graph products, optimality is asserted using the "Converse to the Expander Mixing Lemma" obtained by Bilu and Linial in 2006. A byproduct of our proof technique is a new explicit construction of a family of co-spectral graphs.
△ Less
Submitted 4 September, 2013; v1 submitted 7 November, 2012;
originally announced November 2012.
-
Getting directed Hamilton cycle twice faster
Authors:
Choongbum Lee,
Benny Sudakov,
Dan Vilenchik
Abstract:
Consider the random graph process where we start with an empty graph on n vertices, and at time t, are given an edge e_t chosen uniformly at random among the edges which have not appeared so far. A classical result in random graph theory asserts that w.h.p. the graph becomes Hamiltonian at time (1/2+o(1))n log n. On the contrary, if all the edges were directed randomly, then the graph has a direct…
▽ More
Consider the random graph process where we start with an empty graph on n vertices, and at time t, are given an edge e_t chosen uniformly at random among the edges which have not appeared so far. A classical result in random graph theory asserts that w.h.p. the graph becomes Hamiltonian at time (1/2+o(1))n log n. On the contrary, if all the edges were directed randomly, then the graph has a directed Hamilton cycle w.h.p. only at time (1+o(1))n log n. In this paper we further study the directed case, and ask whether it is essential to have twice as many edges compared to the undirected case. More precisely, we ask if at time t, instead of a random direction one is allowed to choose the orientation of e_t, then whether it is possible or not to make the resulting directed graph Hamiltonian at time earlier than n log n. The main result of our paper answers this question in the strongest possible way, by asserting that one can orient the edges on-line so that w.h.p., the resulting graph has a directed Hamilton cycle exactly at the time at which the underlying graph is Hamiltonian.
△ Less
Submitted 29 March, 2012; v1 submitted 28 March, 2011;
originally announced March 2011.
-
On the diameter of the set of satisfying assignments in random satisfiable k-CNF formulas
Authors:
Uriel Feige,
Abraham D. Flaxman,
Dan Vilenchik
Abstract:
It is known that random k-CNF formulas have a so-called satisfiability threshold at a density (namely, clause-variable ratio) of roughly 2^k\ln 2: at densities slightly below this threshold almost all k-CNF formulas are satisfiable whereas slightly above this threshold almost no k-CNF formula is satisfiable. In the current work we consider satisfiable random formulas, and inspect another paramet…
▽ More
It is known that random k-CNF formulas have a so-called satisfiability threshold at a density (namely, clause-variable ratio) of roughly 2^k\ln 2: at densities slightly below this threshold almost all k-CNF formulas are satisfiable whereas slightly above this threshold almost no k-CNF formula is satisfiable. In the current work we consider satisfiable random formulas, and inspect another parameter -- the diameter of the solution space (that is the maximal Hamming distance between a pair of satisfying assignments).
It was previously shown that for all densities up to a density slightly below the satisfiability threshold the diameter is almost surely at least roughly n/2 (and n at much lower densities). At densities very much higher than the satisfiability threshold, the diameter is almost surely zero (a very dense satisfiable formula is expected to have only one satisfying assignment). In this paper we show that for all densities above a density that is slightly above the satisfiability threshold (more precisely at ratio (1+ \eps)2^k \ln 2, \eps=\eps(k) tending to 0 as k grows) the diameter is almost surely O(k2^{-k}n). This shows that a relatively small change in the density around the satisfiability threshold (a multiplicative (1 + \eps) factor), makes a dramatic change in the diameter. This drop in the diameter cannot be attributed to the fact that a larger fraction of the formulas is not satisfiable (and hence have diameter 0), because the non-satisfiable formulas are excluded from consideration by our conditioning that the formula is satisfiable.
△ Less
Submitted 11 February, 2009;
originally announced February 2009.
-
Complete convergence of message passing algorithms for some satisfiability problems
Authors:
Uriel Feige,
Elchanan Mossel,
Dan Vilenchik
Abstract:
In this paper we analyze the performance of Warning Propagation, a popular message passing algorithm. We show that for 3CNF formulas drawn from a certain distribution over random satisfiable 3CNF formulas, commonly referred to as the planted-assignment distribution, running Warning Propagation in the standard way (run message passing until convergence, simplify the formula according to the resulti…
▽ More
In this paper we analyze the performance of Warning Propagation, a popular message passing algorithm. We show that for 3CNF formulas drawn from a certain distribution over random satisfiable 3CNF formulas, commonly referred to as the planted-assignment distribution, running Warning Propagation in the standard way (run message passing until convergence, simplify the formula according to the resulting assignment, and satisfy the remaining subformula, if necessary, using a simple "off the shelf" heuristic) results in a satisfying assignment when the clause-variable ratio is a sufficiently large constant.
△ Less
Submitted 28 December, 2010; v1 submitted 30 November, 2008;
originally announced December 2008.
-
On the random satisfiable process
Authors:
Michael Krivelevich,
Benny Sudakov,
Dan Vilenchik
Abstract:
In this work we suggest a new model for generating random satisfiable k-CNF formulas. To generate such formulas -- randomly permute all 2^k\binom{n}{k} possible clauses over the variables x_1, ..., x_n, and starting from the empty formula, go over the clauses one by one, including each new clause as you go along if after its addition the formula remains satisfiable. We study the evolution of thi…
▽ More
In this work we suggest a new model for generating random satisfiable k-CNF formulas. To generate such formulas -- randomly permute all 2^k\binom{n}{k} possible clauses over the variables x_1, ..., x_n, and starting from the empty formula, go over the clauses one by one, including each new clause as you go along if after its addition the formula remains satisfiable. We study the evolution of this process, namely the distribution over formulas obtained after scanning through the first m clauses (in the random permutation's order).
Random processes with conditioning on a certain property being respected are widely studied in the context of graph properties. This study was pioneered by Ruciński and Wormald in 1992 for graphs with a fixed degree sequence, and also by Erdős, Suen, and Winkler in 1995 for triangle-free and bipartite graphs. Since then many other graph properties were studied such as planarity and H-freeness. Thus our model is a natural extension of this approach to the satisfiability setting.
Our main contribution is as follows. For m \geq cn, c=c(k) a sufficiently large constant, we are able to characterize the structure of the solution space of a typical formula in this distribution. Specifically, we show that typically all satisfying assignments are essentially clustered in one cluster, and all but e^{-Ω(m/n)} n of the variables take the same value in all satisfying assignments. We also describe a polynomial time algorithm that finds with high probability a satisfying assignment for such formulas.
△ Less
Submitted 27 July, 2008;
originally announced July 2008.
-
Message passing for the coloring problem: Gallager meets Alon and Kahale
Authors:
Sonny Ben-Shimon,
Dan Vilenchik
Abstract:
Message passing algorithms are popular in many combinatorial optimization problems. For example, experimental results show that {\em survey propagation} (a certain message passing algorithm) is effective in finding proper $k$-colorings of random graphs in the near-threshold regime. In 1962 Gallager introduced the concept of Low Density Parity Check (LDPC) codes, and suggested a simple decoding a…
▽ More
Message passing algorithms are popular in many combinatorial optimization problems. For example, experimental results show that {\em survey propagation} (a certain message passing algorithm) is effective in finding proper $k$-colorings of random graphs in the near-threshold regime. In 1962 Gallager introduced the concept of Low Density Parity Check (LDPC) codes, and suggested a simple decoding algorithm based on message passing. In 1994 Alon and Kahale exhibited a coloring algorithm and proved its usefulness for finding a $k$-coloring of graphs drawn from a certain planted-solution distribution over $k$-colorable graphs. In this work we show an interpretation of Alon and Kahale's coloring algorithm in light of Gallager's decoding algorithm, thus showing a connection between the two problems - coloring and decoding. This also provides a rigorous evidence for the usefulness of the message passing paradigm for the graph coloring problem. Our techniques can be applied to several other combinatorial optimization problems and networking-related issues.
△ Less
Submitted 21 October, 2007;
originally announced October 2007.
