-
Stochastic Dynamics of Noisy Average Consensus: Analysis and Optimization
Authors:
Tadashi Wadayama,
Ayano Nakai-Kasai
Abstract:
A continuous-time average consensus system is a linear dynamical system defined over a graph, where each node has its own state value that evolves according to a simultaneous linear differential equation. A node is allowed to interact with neighboring nodes. Average consensus is a phenomenon that the all the state values converge to the average of the initial state values. In this paper, we assume…
▽ More
A continuous-time average consensus system is a linear dynamical system defined over a graph, where each node has its own state value that evolves according to a simultaneous linear differential equation. A node is allowed to interact with neighboring nodes. Average consensus is a phenomenon that the all the state values converge to the average of the initial state values. In this paper, we assume that a node can communicate with neighboring nodes through an additive white Gaussian noise channel. We first formulate the noisy average consensus system by using a stochastic differential equation (SDE), which allows us to use the Euler-Maruyama method, a numerical technique for solving SDEs. By studying the stochastic behavior of the residual error of the Euler-Maruyama method, we arrive at the covariance evolution equation. The analysis of the residual error leads to a compact formula for mean squared error (MSE), which shows that the sum of the inverse eigenvalues of the Laplacian matrix is the most dominant factor influencing the MSE. Furthermore, we propose optimization problems aimed at minimizing the MSE at a given target time, and introduce a deep unfolding-based optimization method to solve these problems. The quality of the solution is validated by numerical experiments.
△ Less
Submitted 29 March, 2023;
originally announced March 2023.
-
Chebyshev Inertial Landweber Algorithm for Linear Inverse Problems
Authors:
Tadashi Wadayama,
Satoshi Takabe
Abstract:
The Landweber algorithm defined on complex/real Hilbert spaces is a gradient descent algorithm for linear inverse problems. Our contribution is to present a novel method for accelerating convergence of the Landweber algorithm. In this paper, we first extend the theory of the Chebyshev inertial iteration to the Landweber algorithm on Hilbert spaces. An upper bound on the convergence rate clarifies…
▽ More
The Landweber algorithm defined on complex/real Hilbert spaces is a gradient descent algorithm for linear inverse problems. Our contribution is to present a novel method for accelerating convergence of the Landweber algorithm. In this paper, we first extend the theory of the Chebyshev inertial iteration to the Landweber algorithm on Hilbert spaces. An upper bound on the convergence rate clarifies the speed of global convergence of the proposed method. The Chebyshev inertial Landweber algorithm can be applied to wide class of signal recovery problems on a Hilbert space including deconvolution for continuous signals. The theoretical discussion developed in this paper naturally leads to a novel practical signal recovery algorithm. As a demonstration, a MIMO detection algorithm based on the projected Landweber algorithm is derived. The proposed MIMO detection algorithm achieves much smaller symbol error rate compared with the MMSE detector.
△ Less
Submitted 16 January, 2020;
originally announced January 2020.
-
Theoretical Interpretation of Learned Step Size in Deep-Unfolded Gradient Descent
Authors:
Satoshi Takabe,
Tadashi Wadayama
Abstract:
Deep unfolding is a promising deep-learning technique in which an iterative algorithm is unrolled to a deep network architecture with trainable parameters. In the case of gradient descent algorithms, as a result of the training process, one often observes the acceleration of the convergence speed with learned non-constant step size parameters whose behavior is not intuitive nor interpretable from…
▽ More
Deep unfolding is a promising deep-learning technique in which an iterative algorithm is unrolled to a deep network architecture with trainable parameters. In the case of gradient descent algorithms, as a result of the training process, one often observes the acceleration of the convergence speed with learned non-constant step size parameters whose behavior is not intuitive nor interpretable from conventional theory. In this paper, we provide a theoretical interpretation of the learned step size of deep-unfolded gradient descent (DUGD). We first prove that the training process of DUGD reduces not only the mean squared error loss but also the spectral radius related to the convergence rate. Next, we show that minimizing the upper bound of the spectral radius naturally leads to the Chebyshev step which is a sequence of the step size based on Chebyshev polynomials. The numerical experiments confirm that the Chebyshev steps qualitatively reproduce the learned step size parameters in DUGD, which provides a plausible interpretation of the learned parameters. Additionally, we show that the Chebyshev steps achieve the lower bound of the convergence rate for the first-order method in a specific limit without learning parameters or momentum terms.
△ Less
Submitted 30 January, 2020; v1 submitted 15 January, 2020;
originally announced January 2020.
-
Chebyshev Inertial Iteration for Accelerating Fixed-Point Iterations
Authors:
Tadashi Wadayama,
Satoshi Takabe
Abstract:
A novel method which is called the Chebyshev inertial iteration for accelerating the convergence speed of fixed-point iterations is presented. The Chebyshev inertial iteration can be regarded as a valiant of the successive over relaxation or Krasnosel'skiǐ-Mann iteration utilizing the inverse of roots of a Chebyshev polynomial as iteration dependent inertial factors. One of the most notable featur…
▽ More
A novel method which is called the Chebyshev inertial iteration for accelerating the convergence speed of fixed-point iterations is presented. The Chebyshev inertial iteration can be regarded as a valiant of the successive over relaxation or Krasnosel'skiǐ-Mann iteration utilizing the inverse of roots of a Chebyshev polynomial as iteration dependent inertial factors. One of the most notable features of the proposed method is that it can be applied to nonlinear fixed-point iterations in addition to linear fixed-point iterations. Linearization around the fixed point is the key for the analysis on the local convergence rate of the proposed method. The proposed method appears effective in particular for accelerating the proximal gradient methods such as ISTA. It is also proved that the proposed method can successfully accelerate almost any fixed-point iterations if all the eigenvalues of the Jacobian at the fixed point are real.
△ Less
Submitted 9 January, 2020;
originally announced January 2020.
-
Deep Learning-Based Average Consensus
Authors:
Masako Kishida,
Masaki Ogura,
Yuichi Yoshida,
Tadashi Wadayama
Abstract:
In this study, we analyzed the problem of accelerating the linear average consensus algorithm for complex networks. We propose a data-driven approach to tuning the weights of temporal (i.e., time-varying) networks using deep learning techniques. Given a finite-time window, the proposed approach first unfolds the linear average consensus protocol to obtain a feedforward signal-flow graph, which is…
▽ More
In this study, we analyzed the problem of accelerating the linear average consensus algorithm for complex networks. We propose a data-driven approach to tuning the weights of temporal (i.e., time-varying) networks using deep learning techniques. Given a finite-time window, the proposed approach first unfolds the linear average consensus protocol to obtain a feedforward signal-flow graph, which is regarded as a neural network. The edge weights of the obtained neural network are then trained using standard deep learning techniques to minimize consensus error over a given finite-time window. Through this training process, we obtain a set of optimized time-varying weights, which yield faster consensus for a complex network. We also demonstrate that the proposed approach can be extended for infinite-time window problems. Numerical experiments revealed that our approach can achieve a significantly smaller consensus error compared to baseline strategies.
△ Less
Submitted 6 August, 2020; v1 submitted 26 August, 2019;
originally announced August 2019.
-
LP Decodable Permutation Codes based on Linearly Constrained Permutation Matrices
Authors:
Tadashi Wadayama,
Manabu Hagiwara
Abstract:
A set of linearly constrained permutation matrices are proposed for constructing a class of permutation codes. Making use of linear constraints imposed on the permutation matrices, we can formulate a minimum Euclidian distance decoding problem for the proposed class of permutation codes as a linear programming (LP) problem. The main feature of this class of permutation codes, called LP decodable p…
▽ More
A set of linearly constrained permutation matrices are proposed for constructing a class of permutation codes. Making use of linear constraints imposed on the permutation matrices, we can formulate a minimum Euclidian distance decoding problem for the proposed class of permutation codes as a linear programming (LP) problem. The main feature of this class of permutation codes, called LP decodable permutation codes, is this LP decodability. It is demonstrated that the LP decoding performance of the proposed class of permutation codes is characterized by the vertices of the code polytope of the code. Two types of linear constraints are discussed; one is structured constraints and another is random constraints. The structured constraints such as pure involution lead to an efficient encoding algorithm. On the other hand, the random constraints enable us to use probabilistic methods for analyzing several code properties such as the average cardinality and the average weight distribution.
△ Less
Submitted 8 July, 2011; v1 submitted 29 November, 2010;
originally announced November 2010.
-
A typical reconstruction limit of compressed sensing based on Lp-norm minimization
Authors:
Y. Kabashima,
T. Wadayama,
T. Tanaka
Abstract:
We consider the problem of reconstructing an $N$-dimensional continuous vector $\bx$ from $P$ constraints which are generated by its linear transformation under the assumption that the number of non-zero elements of $\bx$ is typically limited to $ρN$ ($0\le ρ\le 1$). Problems of this type can be solved by minimizing a cost function with respect to the $L_p$-norm…
▽ More
We consider the problem of reconstructing an $N$-dimensional continuous vector $\bx$ from $P$ constraints which are generated by its linear transformation under the assumption that the number of non-zero elements of $\bx$ is typically limited to $ρN$ ($0\le ρ\le 1$). Problems of this type can be solved by minimizing a cost function with respect to the $L_p$-norm $||\bx||_p=\lim_{ε\to +0}\sum_{i=1}^N |x_i|^{p+ε}$, subject to the constraints under an appropriate condition. For several $p$, we assess a typical case limit $α_c(ρ)$, which represents a critical relation between $α=P/N$ and $ρ$ for successfully reconstructing the original vector by minimization for typical situations in the limit $N,P \to \infty$ with keeping $α$ finite, utilizing the replica method. For $p=1$, $α_c(ρ)$ is considerably smaller than its worst case counterpart, which has been rigorously derived by existing literature of information theory.
△ Less
Submitted 5 December, 2009; v1 submitted 6 July, 2009;
originally announced July 2009.
