Showing 1–4 of 4 results for author: Khalafi, M

Regularized Operator Extrapolation Method For Stochastic Bilevel Variational Inequality Problems

Authors: Mohammad Khalafi, Digvijay Boob

Abstract: The bilevel variational inequality (BVI) problem is a general model that captures various optimization problems, including VI-constrained optimization and equilibrium problems with equilibrium constraints (EPECs). This paper introduces a first-order method for smooth or nonsmooth BVI with stochastic monotone operators at inner and outer levels. Our novel method, called Regularized Operator Extra… ▽ More The bilevel variational inequality (BVI) problem is a general model that captures various optimization problems, including VI-constrained optimization and equilibrium problems with equilibrium constraints (EPECs). This paper introduces a first-order method for smooth or nonsmooth BVI with stochastic monotone operators at inner and outer levels. Our novel method, called Regularized Operator Extrapolation $(\texttt{R-OpEx})$, is a single-loop algorithm that combines Tikhonov's regularization with operator extrapolation. This method needs only one operator evaluation for each operator per iteration and tracks one sequence of iterates. We show that $\texttt{R-OpEx}$ gives $\mathcal{O}(ε^{-4})$ complexity in nonsmooth stochastic monotone BVI, where $ε$ is the error in the inner and outer levels. Using a mini-batching scheme, we improve the outer level complexity to $\mathcal{O}(ε^{-2})$ while maintaining the $\mathcal{O}(ε^{-4})$ complexity in the inner level when the inner level is smooth and stochastic. Moreover, if the inner level is smooth and deterministic, we show complexity of $\mathcal{O}(ε^{-2})$. Finally, in case the outer level is strongly monotone, we improve to $\mathcal{O}(ε^{-4/5})$ for general BVI and $\mathcal{O}(ε^{-2/3})$ when the inner level is smooth and deterministic. To our knowledge, this is the first work that investigates nonsmooth stochastic BVI with the best-known convergence guarantees. We verify our theoretical results with numerical experiments. △ Less

Submitted 14 May, 2025; originally announced May 2025.

Optimal Primal-Dual Algorithm with Last iterate Convergence Guarantees for Stochastic Convex Optimization Problems

Authors: Digvijay Boob, Mohammad Khalafi

Abstract: This paper proposes a novel first-order algorithm that solves composite nonsmooth and stochastic convex optimization problem with function constraints. Most of the works in the literature provide convergence rate guarantees on the average-iterate solution. There is growing interest in the convergence guarantees of the last iterate solution due to its favorable structural properties, such as sparsi… ▽ More This paper proposes a novel first-order algorithm that solves composite nonsmooth and stochastic convex optimization problem with function constraints. Most of the works in the literature provide convergence rate guarantees on the average-iterate solution. There is growing interest in the convergence guarantees of the last iterate solution due to its favorable structural properties, such as sparsity or privacy guarantees and good performance in practice. We provide the first method that obtains the best-known convergence rate guarantees on the last iterate for stochastic composite nonsmooth convex function-constrained optimization problems. Our novel and easy-to-implement algorithm is based on the augmented Lagrangian technique and uses a new linearized approximation of constraint functions, leading to its name, the Augmented Constraint Extrapolation (Aug-ConEx) method. We show that Aug-ConEx achieves $\mathcal{O}(1/\sqrt{K})$ convergence rate in the nonsmooth stochastic setting without any strong convexity assumption and $\mathcal{O}(1/K)$ for the same problem with strongly convex objective function. While optimal for nonsmooth and stochastic problems, the Aug-ConEx method also accelerates convergence in terms of Lipschitz smoothness constants to $\mathcal{O}(1/K)$ and $\mathcal{O}(1/K^2)$ in the aforementioned cases, respectively. To our best knowledge, this is the first method to obtain such differentiated convergence rate guarantees on the last iterate for a composite nonsmooth stochastic setting without additional $\log{K}$ factors. We validate the efficiency of our algorithm by comparing it with a state-of-the-art algorithm through numerical experiments. △ Less

Submitted 24 October, 2024; originally announced October 2024.

Comments: 26 Pages, 3 Figures

First-order methods for Stochastic Variational Inequality problems with Function Constraints

Authors: Digvijay Boob, Qi Deng, Mohammad Khalafi

Abstract: The monotone Variational Inequality (VI) is a general model with important applications in various engineering and scientific domains. In numerous instances, the VI problems are accompanied by function constraints that can be data-driven, making the usual projection operator challenging to compute. This paper presents novel first-order methods for the function-constrained Variational Inequality (F… ▽ More The monotone Variational Inequality (VI) is a general model with important applications in various engineering and scientific domains. In numerous instances, the VI problems are accompanied by function constraints that can be data-driven, making the usual projection operator challenging to compute. This paper presents novel first-order methods for the function-constrained Variational Inequality (FCVI) problem in smooth or nonsmooth settings with possibly stochastic operators and constraints. We introduce the AdOpEx method, which employs an operator extrapolation on the KKT operator of the FCVI in a smooth deterministic setting. Since this operator is not uniformly Lipschitz continuous in the Lagrange multipliers, we employ an adaptive two-timescale algorithm leading to bounded multipliers and achieving the optimal $O(1/T)$ convergence rate. For the nonsmooth and stochastic VIs, we introduce design changes to the AdOpEx method and propose a novel P-OpEx method that takes partial extrapolation. It converges at the rate of $O(1/\sqrt{T})$ when both the operator and constraints are stochastic or nonsmooth. This method has suboptimal dependence on the noise and Lipschitz constants of function constraints. We propose a constraint extrapolation approach leading to the OpConEx method that improves this dependence by an order of magnitude. All our algorithms easily extend to saddle point problems with function constraints that couple the primal and dual variables while maintaining the same complexity results. To the best of our knowledge, all our complexity results are new in the literature △ Less

Submitted 24 May, 2024; v1 submitted 10 April, 2023; originally announced April 2023.

Accelerated Primal-Dual Methods for Convex-Strongly-Concave Saddle Point Problems

Authors: Mohammad Khalafi, Digvijay Boob

Abstract: We investigate a primal-dual (PD) method for the saddle point problem (SPP) that uses a linear approximation of the primal function instead of the standard proximal step, resulting in a linearized PD (LPD) method. For convex-strongly concave SPP, we observe that the LPD method has a suboptimal dependence on the Lipschitz constant of the primal function. To fix this issue, we combine features of Ac… ▽ More We investigate a primal-dual (PD) method for the saddle point problem (SPP) that uses a linear approximation of the primal function instead of the standard proximal step, resulting in a linearized PD (LPD) method. For convex-strongly concave SPP, we observe that the LPD method has a suboptimal dependence on the Lipschitz constant of the primal function. To fix this issue, we combine features of Accelerated Gradient Descent with the LPD method resulting in a single-loop Accelerated Linearized Primal-Dual (ALPD) method. ALPD method achieves the optimal gradient complexity when the SPP has a semi-linear coupling function. We also present an inexact ALPD method for SPPs with a general nonlinear coupling function that maintains the optimal gradient evaluations of the primal parts and significantly improves the gradient evaluations of the coupling term compared to the ALPD method. We verify our findings with numerical experiments. △ Less

Submitted 18 May, 2023; v1 submitted 10 September, 2022; originally announced September 2022.