Title:Provable Accelerated Gradient Method for Nonconvex Low Rank Optimization

Abstract:Optimization with a low rank constraint has broad applications. For large scale problems, an attractive heuristic is to factorize the low rank matrix to a product of two much smaller matrices. In this paper, we study the nonconvex problem $\min_{U\in\mathbf R^{n\times r}} g(U)=f(UU^T)$ under the assumptions that $f(X)$ is restricted $\mu$-strongly convex and $L$-smooth on the set $\{X:X\succeq 0,\mbox{rank}(X)\leq r\}$. We propose an accelerated gradient method with the provable convergence rate of $g(U^{N+1})-g(U^*)\leq \left(1-O\left(\sqrt{\frac{\mu}{nL}}\right)\right)^N\|U^0-U^*\|_F^2$, where $U^*$ can be any local minimum that the algorithm converges to. The convergence rate can be improved to $\left(1-O\left(\sqrt{\frac{\mu}{L}}\right)\right)^N$ under a practical assumption that the entries of $U^{k}$ converge nearly equally fast to those of $U^*$. To the best of our knowledge, this is the first work establishing the linear convergence with the optimal dependence on the condition number $L/\mu$ for this kind of nonconvex problems. Our method can also be transferred to the problem with the asymmetric factorization of $X=\widetilde U\widetilde V^T$.