-
Gossip PCA
Authors:
Satish Babu Korada,
Andrea Montanari,
Sewoong Oh
Abstract:
Eigenvectors of data matrices play an important role in many computational problems, ranging from signal processing to machine learning and control. For instance, algorithms that compute positions of the nodes of a wireless network on the basis of pairwise distance measurements require a few leading eigenvectors of the distances matrix. While eigenvector calculation is a standard topic in numerica…
▽ More
Eigenvectors of data matrices play an important role in many computational problems, ranging from signal processing to machine learning and control. For instance, algorithms that compute positions of the nodes of a wireless network on the basis of pairwise distance measurements require a few leading eigenvectors of the distances matrix. While eigenvector calculation is a standard topic in numerical linear algebra, it becomes challenging under severe communication or computation constraints, or in absence of central scheduling. In this paper we investigate the possibility of computing the leading eigenvectors of a large data matrix through gossip algorithms.
The proposed algorithm amounts to iteratively multiplying a vector by independent random sparsification of the original matrix and averaging the resulting normalized vectors. This can be viewed as a generalization of gossip algorithms for consensus, but the resulting dynamics is significantly more intricate. Our analysis is based on controlling the convergence to stationarity of the associated Kesten-Furstenberg Markov chain.
△ Less
Submitted 21 March, 2011;
originally announced March 2011.
-
Applications of Lindeberg Principle in Communications and Statistical Learning
Authors:
Satish Babu Korada,
Andrea Montanari
Abstract:
We use a generalization of the Lindeberg principle developed by Sourav Chatterjee to prove universality properties for various problems in communications, statistical learning and random matrix theory. We also show that these systems can be viewed as the limiting case of a properly defined sparse system. The latter result is useful when the sparse systems are easier to analyze than their dense co…
▽ More
We use a generalization of the Lindeberg principle developed by Sourav Chatterjee to prove universality properties for various problems in communications, statistical learning and random matrix theory. We also show that these systems can be viewed as the limiting case of a properly defined sparse system. The latter result is useful when the sparse systems are easier to analyze than their dense counterparts. The list of problems we consider is by no means exhaustive. We believe that the ideas can be used in many other problems relevant for information theory.
△ Less
Submitted 7 April, 2010; v1 submitted 4 April, 2010;
originally announced April 2010.
-
The Compound Capacity of Polar Codes
Authors:
S. Hamed Hassani,
Satish Babu Korada,
Ruediger Urbanke
Abstract:
We consider the compound capacity of polar codes under successive cancellation decoding for a collection of binary-input memoryless output-symmetric channels. By deriving a sequence of upper and lower bounds, we show that in general the compound capacity under successive decoding is strictly smaller than the unrestricted compound capacity.
We consider the compound capacity of polar codes under successive cancellation decoding for a collection of binary-input memoryless output-symmetric channels. By deriving a sequence of upper and lower bounds, we show that in general the compound capacity under successive decoding is strictly smaller than the unrestricted compound capacity.
△ Less
Submitted 19 July, 2009;
originally announced July 2009.
-
Polar Codes are Optimal for Lossy Source Coding
Authors:
Satish Babu Korada,
Rudiger Urbanke
Abstract:
We consider lossy source compression of a binary symmetric source using polar codes and the low-complexity successive encoding algorithm. It was recently shown by Arikan that polar codes achieve the capacity of arbitrary symmetric binary-input discrete memoryless channels under a successive decoding strategy. We show the equivalent result for lossy source compression, i.e., we show that this com…
▽ More
We consider lossy source compression of a binary symmetric source using polar codes and the low-complexity successive encoding algorithm. It was recently shown by Arikan that polar codes achieve the capacity of arbitrary symmetric binary-input discrete memoryless channels under a successive decoding strategy. We show the equivalent result for lossy source compression, i.e., we show that this combination achieves the rate-distortion bound for a binary symmetric source. We further show the optimality of polar codes for various problems including the binary Wyner-Ziv and the binary Gelfand-Pinsker problem
△ Less
Submitted 2 March, 2009;
originally announced March 2009.
-
Performance of Polar Codes for Channel and Source Coding
Authors:
Nadine Hussami,
Satish Babu Korada,
Rudiger Urbanke
Abstract:
Polar codes, introduced recently by Arıkan, are the first family of codes known to achieve capacity of symmetric channels using a low complexity successive cancellation decoder. Although these codes, combined with successive cancellation, are optimal in this respect, their finite-length performance is not record breaking. We discuss several techniques through which their finite-length performanc…
▽ More
Polar codes, introduced recently by Arıkan, are the first family of codes known to achieve capacity of symmetric channels using a low complexity successive cancellation decoder. Although these codes, combined with successive cancellation, are optimal in this respect, their finite-length performance is not record breaking. We discuss several techniques through which their finite-length performance can be improved. We also study the performance of these codes in the context of source coding, both lossless and lossy, in the single-user context as well as for distributed applications.
△ Less
Submitted 22 May, 2009; v1 submitted 16 January, 2009;
originally announced January 2009.
-
Polar Codes: Characterization of Exponent, Bounds, and Constructions
Authors:
Satish Babu Korada,
Eren Sasoglu,
Rudiger Urbanke
Abstract:
Polar codes were recently introduced by Arıkan. They achieve the capacity of arbitrary symmetric binary-input discrete memoryless channels under a low complexity successive cancellation decoding strategy. The original polar code construction is closely related to the recursive construction of Reed-Muller codes and is based on the $2 \times 2$ matrix $\bigl[ 1 &0 1& 1 \bigr]$. It was shown by Arı…
▽ More
Polar codes were recently introduced by Arıkan. They achieve the capacity of arbitrary symmetric binary-input discrete memoryless channels under a low complexity successive cancellation decoding strategy. The original polar code construction is closely related to the recursive construction of Reed-Muller codes and is based on the $2 \times 2$ matrix $\bigl[ 1 &0 1& 1 \bigr]$. It was shown by Arıkan and Telatar that this construction achieves an error exponent of $\frac12$, i.e., that for sufficiently large blocklengths the error probability decays exponentially in the square root of the length. It was already mentioned by Arıkan that in principle larger matrices can be used to construct polar codes. A fundamental question then is to see whether there exist matrices with exponent exceeding $\frac12$. We first show that any $\ell \times \ell$ matrix none of whose column permutations is upper triangular polarizes symmetric channels. We then characterize the exponent of a given square matrix and derive upper and lower bounds on achievable exponents. Using these bounds we show that there are no matrices of size less than 15 with exponents exceeding $\frac12$. Further, we give a general construction based on BCH codes which for large $n$ achieves exponents arbitrarily close to 1 and which exceeds $\frac12$ for size 16.
△ Less
Submitted 26 January, 2009; v1 submitted 5 January, 2009;
originally announced January 2009.
-
A Class of Transformations that Polarize Symmetric Binary-Input Memoryless Channels
Authors:
Satish Babu Korada,
Eren Sasoglu
Abstract:
A generalization of Arıkan's polar code construction using transformations of the form $G^{\otimes n}$ where $G$ is an $\ell \times \ell$ matrix is considered. Necessary and sufficient conditions are given for these transformations to ensure channel polarization. It is shown that a large class of such transformations polarize symmetric binary-input memoryless channels.
A generalization of Arıkan's polar code construction using transformations of the form $G^{\otimes n}$ where $G$ is an $\ell \times \ell$ matrix is considered. Necessary and sufficient conditions are given for these transformations to ensure channel polarization. It is shown that a large class of such transformations polarize symmetric binary-input memoryless channels.
△ Less
Submitted 11 November, 2008;
originally announced November 2008.
-
Tight Bounds on the Capacity of Binary Input random CDMA Systems
Authors:
Satish Babu Korada,
Nicolas Macris
Abstract:
We consider multiple access communication on a binary input additive white Gaussian noise channel using randomly spread code division. For a general class of symmetric distributions for spreading coefficients, in the limit of a large number of users, we prove an upper bound on the capacity, which matches a formula that Tanaka obtained by using the replica method. We also show concentration of va…
▽ More
We consider multiple access communication on a binary input additive white Gaussian noise channel using randomly spread code division. For a general class of symmetric distributions for spreading coefficients, in the limit of a large number of users, we prove an upper bound on the capacity, which matches a formula that Tanaka obtained by using the replica method. We also show concentration of various relevant quantities including mutual information, capacity and free energy. The mathematical methods are quite general and allow us to discuss extensions to other multiuser scenarios.
△ Less
Submitted 10 March, 2008;
originally announced March 2008.
-
Exchange of Limits: Why Iterative Decoding Works
Authors:
Satish Babu Korada,
Ruediger Urbanke
Abstract:
We consider communication over binary-input memoryless output-symmetric channels using low-density parity-check codes and message-passing decoding. The asymptotic (in the length) performance of such a combination for a fixed number of iterations is given by density evolution. Letting the number of iterations tend to infinity we get the density evolution threshold, the largest channel parameter s…
▽ More
We consider communication over binary-input memoryless output-symmetric channels using low-density parity-check codes and message-passing decoding. The asymptotic (in the length) performance of such a combination for a fixed number of iterations is given by density evolution. Letting the number of iterations tend to infinity we get the density evolution threshold, the largest channel parameter so that the bit error probability tends to zero as a function of the iterations.
In practice we often work with short codes and perform a large number of iterations. It is therefore interesting to consider what happens if in the standard analysis we exchange the order in which the blocklength and the number of iterations diverge to infinity. In particular, we can ask whether both limits give the same threshold.
Although empirical observations strongly suggest that the exchange of limits is valid for all channel parameters, we limit our discussion to channel parameters below the density evolution threshold. Specifically, we show that under some suitable technical conditions the bit error probability vanishes below the density evolution threshold regardless of how the limit is taken.
△ Less
Submitted 10 February, 2008;
originally announced February 2008.
