Showing 1–3 of 3 results for author: Boroun, M

Projection-Free Methods for Solving Nonconvex-Concave Saddle Point Problems

Authors: Morteza Boroun, Erfan Yazdandoost Hamedani, Afrooz Jalilzadeh

Abstract: In this paper, we investigate a class of constrained saddle point (SP) problems where the objective function is nonconvex-concave and smooth. This class of problems has wide applicability in machine learning, including robust multi-class classification and dictionary learning. Several projection-based primal-dual methods have been developed for tackling this problem; however, the availability of m… ▽ More In this paper, we investigate a class of constrained saddle point (SP) problems where the objective function is nonconvex-concave and smooth. This class of problems has wide applicability in machine learning, including robust multi-class classification and dictionary learning. Several projection-based primal-dual methods have been developed for tackling this problem; however, the availability of methods with projection-free oracles remains limited. To address this gap, we propose efficient single-loop projection-free methods reliant on first-order information. In particular, using regularization and nested approximation techniques, we propose a primal-dual conditional gradient method that solely employs linear minimization oracles to handle constraints. Assuming that the constraint set in the maximization is strongly convex, our method achieves an $ε$-stationary solution within $\mathcal{O}(ε^{-6})$ iterations. When the projection onto the constraint set of maximization is easy to compute, we propose a one-sided projection-free method that achieves an $ε$-stationary solution within $\mathcal{O}(ε^{-4})$ iterations. Moreover, we present improved iteration complexities of our methods under a strong concavity assumption. To the best of our knowledge, our proposed algorithms are among the first projection-free methods with convergence guarantees for solving nonconvex-concave SP problems. △ Less

Submitted 1 November, 2023; v1 submitted 20 June, 2023; originally announced June 2023.

Comments: Accepted to NeurIPS 2023

Accelerated Primal-dual Scheme for a Class of Stochastic Nonconvex-concave Saddle Point Problems

Authors: Morteza Boroun, Zeinab Alizadeh, Afrooz Jalilzadeh

Abstract: Stochastic nonconvex-concave min-max saddle point problems appear in many machine learning and control problems including distributionally robust optimization, generative adversarial networks, and adversarial learning. In this paper, we consider a class of nonconvex saddle point problems where the objective function satisfies the Polyak-Łojasiewicz condition with respect to the minimization variab… ▽ More Stochastic nonconvex-concave min-max saddle point problems appear in many machine learning and control problems including distributionally robust optimization, generative adversarial networks, and adversarial learning. In this paper, we consider a class of nonconvex saddle point problems where the objective function satisfies the Polyak-Łojasiewicz condition with respect to the minimization variable and it is concave with respect to the maximization variable. The existing methods for solving nonconvex-concave saddle point problems often suffer from slow convergence and/or contain multiple loops. Our main contribution lies in proposing a novel single-loop accelerated primal-dual algorithm with new convergence rate results appearing for the first time in the literature, to the best of our knowledge. In particular, in the stochastic regime, we demonstrate a convergence rate of $\mathcal O(ε^{-4})$ to find an $ε$-gap solution which can be improved to $\mathcal O(ε^{-2})$ in deterministic setting. △ Less

Submitted 11 September, 2023; v1 submitted 28 February, 2023; originally announced March 2023.

Inexact-Proximal Accelerated Gradient Method for Stochastic Nonconvex Constrained Optimization Problems

Authors: Morteza Boroun, Afrooz Jalilzadeh

Abstract: Stochastic nonconvex optimization problems with nonlinear constraints have a broad range of applications in intelligent transportation, cyber-security, and smart grids. In this paper, first, we propose an inexact-proximal accelerated gradient method to solve a nonconvex stochastic composite optimization problem where the objective is the sum of smooth and nonsmooth functions, the constraint functi… ▽ More Stochastic nonconvex optimization problems with nonlinear constraints have a broad range of applications in intelligent transportation, cyber-security, and smart grids. In this paper, first, we propose an inexact-proximal accelerated gradient method to solve a nonconvex stochastic composite optimization problem where the objective is the sum of smooth and nonsmooth functions, the constraint functions are assumed to be deterministic and the solution to the proximal map of the nonsmooth part is calculated inexactly at each iteration. We demonstrate an asymptotic sublinear rate of convergence for stochastic settings using increasing sample-size considering the error in the proximal operator diminishes at an appropriate rate. Then we customize the proposed method for solving stochastic nonconvex optimization problems with nonlinear constraints and demonstrate a convergence rate guarantee. Numerical results show the effectiveness of the proposed algorithm. △ Less

Submitted 7 July, 2021; v1 submitted 15 April, 2021; originally announced April 2021.