Title:A Hybrid Method of Combinatorial Search and Gradient Descent for Discrete Optimization

Abstract:Discrete optimization is a central problem in machine learning with a broad range of applications, among which binary optimization and sparse optimization are two common ones. However, these problems are NP-hard and thus difficult to solve in general. Combinatorial search methods such as branch-and-bound and exhaustive search find the global optimal solution but are confined to small-sized problems, while gradient descent methods such as proximal gradient descent and coordinate gradient descent are efficient but often suffer from poor local minima. In this paper, we consider a hybrid method that combines the effectiveness of combinatorial search and the efficiency of gradient descent. Specifically, we randomly select a subset of coordinates and then perform global combinatorial search over the small subset of coordinates based on the gradient information of the smooth objective function. The proposed method can be viewed as a randomized block proximal Newton method. In addition, we provide some optimality analysis and convergence analysis for the proposed method. Finally, we demonstrate the efficacy of our method on some sparse optimization and binary optimization applications. As a result, our method achieves state-of-the-art performance in terms of accuracy.