Title:Low-Rank Approximation via Sparse Frequent Directions

Abstract:In this paper, we describe a new randomized algorithm for low-rank approximation -- Sparse Frequent Directions (SpFD). It combines techniques of frequent directions (FD) and sparse subspace embedding (SpEmb). Given a matrix $A\in\mathbb{R}^{n\times d}$, this new algorithm is able to find a sketch $B\in\mathbb{R}^{\ell\times d}$ and a matrix $V\in\mathbb{R}^{d\times \ell}$ whose columns form an orthonormal basis for the row space of $B$, such that with probability at least $1-\delta$, \[ \norm{A- \Pi_{B}^k(A)}_F^2 \left( = \norm{A-[AV]_kV^T}_F^2\right) \leq \norm{A-A_k}_F^2 + \left(\dfrac{k}{(\ell-k)\delta}+\varepsilon\right)\norm{A}_F^2 \] in only $O(nnz(A)+d\ell k^2/(\varepsilon^2\delta))$ time. By taking $\ell = O(k/(\varepsilon\delta))$, the error bound becomes \[ \norm{A- \Pi_{B}^k(A)}_F^2 \leq \norm{A-A_k}_F^2 + \varepsilon\norm{A}_F^2 \] with running time $O(nnz(A)+dk^3/(\varepsilon^3\delta^2))$. In comparison with FD that runs in $O(nd\ell) = O(ndk/\varepsilon)$ time and SpEmb that runs in $O(nnz(A)+d\ell^2) = O(nnz(A)+dk^4/(\varepsilon^4\delta^2))$ time to reach `relative-error' bound, SpFD is rather efficient and respects the possible sparsity of the input matrix. In terms of accuracy, SpFD requires sketch size $\ell = O(k/(\varepsilon\delta))$ which is linear on $k$, on the same order of FD, while SpEmb requires $\ell = O((k^2+k)/(\varepsilon^2\delta))$. SpFD is able to reach high accuracy with relatively small sketch size. SpFD is a combination of FD and SpEmb in favor of FD's accuracy and SpEmb's efficiency which is supported by theory and experimental results.