Title:Understanding the ADMM Algorithm via High-Resolution Differential Equations

Abstract:In the fields of statistics, machine learning, image science, and related areas, there is an increasing demand for decentralized collection or storage of large-scale datasets, as well as distributed solution methods. To tackle this challenge, the alternating direction method of multipliers (ADMM) has emerged as a widely used approach, particularly well-suited to distributed convex optimization. However, the iterative behavior of ADMM has not been well understood. In this paper, we employ dimensional analysis to derive a system of high-resolution ordinary differential equations (ODEs) for ADMM. This system captures an important characteristic of ADMM, called the $\lambda$-correction, which causes the trajectory of ADMM to deviate from the constrained hyperplane. To explore the convergence behavior of the system of high-resolution ODEs, we utilize Lyapunov analysis and extend our findings to the discrete ADMM algorithm. Through this analysis, we identify that the numerical error resulting from the implicit scheme is a crucial factor that affects the convergence rate and monotonicity in the discrete ADMM algorithm. In addition, we further discover that if one component of the objective function is assumed to be strongly convex, the iterative average of ADMM converges strongly with a rate $O(1/N)$, where $N$ is the number of iterations.