Title:Faster ICA by preconditioning with Hessian approximations

Abstract:Independent Component Analysis (ICA) is a powerful technique for unsupervised data exploration that is widely used across fields such as neuroscience, astronomy, chemistry or biology. Linear ICA is a linear latent factor model, similar to sparse dictionary learning, that aims at discovering statistically independent sources from multivariate observations. ICA is a probabilistic generative model for which inference is classically done by maximum likelihood estimation. Estimating sources by maximum likelihood leads to a smooth non-convex optimization problem where the unknown is a matrix called the separating or unmixing matrix. As the gradient of the likelihood is available in closed form, first order gradient methods, stochastic or non-stochastic, are often employed despite a slow convergence such as in the Infomax algorithm. While the Hessian is known analytically, the cost of its computation and inversion makes Newton method unpractical for a large number of sources. We show how sparse and positive approximations of the true Hessian can be obtained and used to precondition the L-BFGS algorithm. Results on simulations and two applied problems (EEG data and image patches) demonstrate that the proposed technique leads to convergence that can be orders of magnitude faster than algorithms commonly used today even when looking for hundred of sources.