Title:Exact penalty methods for minimizing a smooth function over the nonnegative orthogonal set

Abstract:This paper is concerned with the problem of minimizing a smooth function over the nonnegative orthogonal set. For this class of nonconvex nonsmooth optimization problems, we show that the penalized problem induced by the elementwise $\ell_1$-norm distance to the nonnegative cone is a global exact penalty, and so is the one induced by its Moreau envelop under a growth condition on the objective function. Then, we develop the exact penalty methods by solving the smooth and nonsmooth penalty problems, which are respectively minimizing a smooth function and a composite nonsmooth function over the Steifel manifold in each step. Two retraction-based proximal gradient methods with a line-search strategy are provided for solving the subproblems. Numerical tests on quadratic assignment problems and graph matching problems demonstrate that the penalty method based on the smooth exact penalty is superior to the one based on the nonsmooth exact penalty.