Showing 1–19 of 19 results for author: Jalilzadeh, A

Variance-reduction for Variational Inequality Problems with Bregman Distance Function

Authors: Zeinab Alizadeh, Erfan Yazdandoost Hamedani, Afrooz Jalilzadeh

Abstract: In this paper, we address variational inequalities (VI) with a finite-sum structure. We introduce a novel single-loop stochastic variance-reduced algorithm, incorporating the Bregman distance function, and establish an optimal convergence guarantee under a monotone setting. Additionally, we explore a structured class of non-monotone problems that exhibit weak Minty solutions, and analyze the compl… ▽ More In this paper, we address variational inequalities (VI) with a finite-sum structure. We introduce a novel single-loop stochastic variance-reduced algorithm, incorporating the Bregman distance function, and establish an optimal convergence guarantee under a monotone setting. Additionally, we explore a structured class of non-monotone problems that exhibit weak Minty solutions, and analyze the complexity of our proposed method, highlighting a significant improvement over existing approaches. Numerical experiments are presented to demonstrate the performance of our algorithm compared to state-of-the-art methods △ Less

Submitted 7 January, 2025; v1 submitted 17 May, 2024; originally announced May 2024.

Convergence Analysis of Non-Strongly-Monotone Stochastic Quasi-Variational Inequalities

Authors: Zeinab Alizadeh, Afrooz Jalilzadeh

Abstract: While Variational Inequality (VI) is a well-established mathematical framework that subsumes Nash equilibrium and saddle-point problems, less is known about its extension, Quasi-Variational Inequalities (QVI). QVI allows for cases where the constraint set changes as the decision variable varies allowing for a more versatile setting. In this paper, we propose extra-gradient and gradient-based metho… ▽ More While Variational Inequality (VI) is a well-established mathematical framework that subsumes Nash equilibrium and saddle-point problems, less is known about its extension, Quasi-Variational Inequalities (QVI). QVI allows for cases where the constraint set changes as the decision variable varies allowing for a more versatile setting. In this paper, we propose extra-gradient and gradient-based methods for solving a class of monotone Stochastic Quasi-Variational Inequalities (SQVI) and establish a rigorous convergence rate analysis for these methods. Our approach not only advances the theoretical understanding of SQVI but also demonstrates its practical applicability. Specifically, we highlight its effectiveness in reformulating and solving problems such as generalized Nash Equilibrium, bilevel optimization, and saddle-point problems with coupling constraints. △ Less

Submitted 30 October, 2024; v1 submitted 5 January, 2024; originally announced January 2024.

Projection-Free Methods for Solving Nonconvex-Concave Saddle Point Problems

Authors: Morteza Boroun, Erfan Yazdandoost Hamedani, Afrooz Jalilzadeh

Abstract: In this paper, we investigate a class of constrained saddle point (SP) problems where the objective function is nonconvex-concave and smooth. This class of problems has wide applicability in machine learning, including robust multi-class classification and dictionary learning. Several projection-based primal-dual methods have been developed for tackling this problem; however, the availability of m… ▽ More In this paper, we investigate a class of constrained saddle point (SP) problems where the objective function is nonconvex-concave and smooth. This class of problems has wide applicability in machine learning, including robust multi-class classification and dictionary learning. Several projection-based primal-dual methods have been developed for tackling this problem; however, the availability of methods with projection-free oracles remains limited. To address this gap, we propose efficient single-loop projection-free methods reliant on first-order information. In particular, using regularization and nested approximation techniques, we propose a primal-dual conditional gradient method that solely employs linear minimization oracles to handle constraints. Assuming that the constraint set in the maximization is strongly convex, our method achieves an $ε$-stationary solution within $\mathcal{O}(ε^{-6})$ iterations. When the projection onto the constraint set of maximization is easy to compute, we propose a one-sided projection-free method that achieves an $ε$-stationary solution within $\mathcal{O}(ε^{-4})$ iterations. Moreover, we present improved iteration complexities of our methods under a strong concavity assumption. To the best of our knowledge, our proposed algorithms are among the first projection-free methods with convergence guarantees for solving nonconvex-concave SP problems. △ Less

Submitted 1 November, 2023; v1 submitted 20 June, 2023; originally announced June 2023.

Comments: Accepted to NeurIPS 2023

A Projection-Based Algorithm for Solving Stochastic Inverse Variational Inequality Problems

Authors: Zeinab Alizadeh, Felipe Parra Polanco, Afrooz Jalilzadeh

Abstract: We consider a stochastic Inverse Variational Inequality (IVI) problem defined by a continuous and co-coercive map over a closed and convex set. Motivated by the absence of performance guarantees for stochastic IVI, we present a variance-reduced projection-based gradient method. Our proposed method ensures an almost sure convergence of the generated iterates to the solution, and we establish a conv… ▽ More We consider a stochastic Inverse Variational Inequality (IVI) problem defined by a continuous and co-coercive map over a closed and convex set. Motivated by the absence of performance guarantees for stochastic IVI, we present a variance-reduced projection-based gradient method. Our proposed method ensures an almost sure convergence of the generated iterates to the solution, and we establish a convergence rate guarantee. To verify our results, we apply the proposed algorithm to a network equilibrium control problem. △ Less

Submitted 7 December, 2023; v1 submitted 13 May, 2023; originally announced May 2023.

Randomized Lagrangian Stochastic Approximation for Large-Scale Constrained Stochastic Nash Games

Authors: Zeinab Alizadeh, Afrooz Jalilzadeh, Farzad Yousefian

Abstract: In this paper, we consider stochastic monotone Nash games where each player's strategy set is characterized by possibly a large number of explicit convex constraint inequalities. Notably, the functional constraints of each player may depend on the strategies of other players, allowing for capturing a subclass of generalized Nash equilibrium problems (GNEP). While there is limited work that provide… ▽ More In this paper, we consider stochastic monotone Nash games where each player's strategy set is characterized by possibly a large number of explicit convex constraint inequalities. Notably, the functional constraints of each player may depend on the strategies of other players, allowing for capturing a subclass of generalized Nash equilibrium problems (GNEP). While there is limited work that provide guarantees for this class of stochastic GNEPs, even when the functional constraints of the players are independent of each other, the majority of the existing methods rely on employing projected stochastic approximation (SA) methods. However, the projected SA methods perform poorly when the constraint set is afflicted by the presence of a large number of possibly nonlinear functional inequalities. Motivated by the absence of performance guarantees for computing the Nash equilibrium in constrained stochastic monotone Nash games, we develop a single timescale randomized Lagrangian multiplier stochastic approximation method where in the primal space, we employ an SA scheme, and in the dual space, we employ a randomized block-coordinate scheme where only a randomly selected Lagrangian multiplier is updated. We show that our method achieves a convergence rate of $\mathcal{O}\left(\frac{\log(k)}{\sqrt{k}}\right)$ for suitably defined suboptimality and infeasibility metrics in a mean sense. △ Less

Submitted 23 August, 2023; v1 submitted 16 April, 2023; originally announced April 2023.

Comments: The result of this paper has been presented at International Conference on Continuous Optimization (ICCOPT) 2022 and East Coast Optimization Meeting (ECOM) 2023

Accelerated Primal-dual Scheme for a Class of Stochastic Nonconvex-concave Saddle Point Problems

Authors: Morteza Boroun, Zeinab Alizadeh, Afrooz Jalilzadeh

Abstract: Stochastic nonconvex-concave min-max saddle point problems appear in many machine learning and control problems including distributionally robust optimization, generative adversarial networks, and adversarial learning. In this paper, we consider a class of nonconvex saddle point problems where the objective function satisfies the Polyak-Łojasiewicz condition with respect to the minimization variab… ▽ More Stochastic nonconvex-concave min-max saddle point problems appear in many machine learning and control problems including distributionally robust optimization, generative adversarial networks, and adversarial learning. In this paper, we consider a class of nonconvex saddle point problems where the objective function satisfies the Polyak-Łojasiewicz condition with respect to the minimization variable and it is concave with respect to the maximization variable. The existing methods for solving nonconvex-concave saddle point problems often suffer from slow convergence and/or contain multiple loops. Our main contribution lies in proposing a novel single-loop accelerated primal-dual algorithm with new convergence rate results appearing for the first time in the literature, to the best of our knowledge. In particular, in the stochastic regime, we demonstrate a convergence rate of $\mathcal O(ε^{-4})$ to find an $ε$-gap solution which can be improved to $\mathcal O(ε^{-2})$ in deterministic setting. △ Less

Submitted 11 September, 2023; v1 submitted 28 February, 2023; originally announced March 2023.

An Accelerated Asynchronous Distributed Method for Convex Constrained Optimization Problems

Authors: Nazanin Abolfazli, Afrooz Jalilzadeh, Erfan Yazdandoost Hamedani

Abstract: We consider a class of multi-agent cooperative consensus optimization problems with local nonlinear convex constraints where only those agents connected by an edge can directly communicate, hence, the optimal consensus decision lies in the intersection of these private sets. We develop an asynchronous distributed accelerated primal-dual algorithm to solve the considered problem. The proposed schem… ▽ More We consider a class of multi-agent cooperative consensus optimization problems with local nonlinear convex constraints where only those agents connected by an edge can directly communicate, hence, the optimal consensus decision lies in the intersection of these private sets. We develop an asynchronous distributed accelerated primal-dual algorithm to solve the considered problem. The proposed scheme is the first asynchronous method with an optimal convergence guarantee for this class of problems, to the best of our knowledge. In particular, we provide an optimal convergence rate of $\mathcal{O(1/K)}$ for suboptimality, infeasibility, and consensus violation. △ Less

Submitted 21 February, 2023; originally announced February 2023.

An Inexact Variance-Reduced Method For Stochastic Quasi-Variational Inequality Problems With An Application In Healthcare

Authors: Zeinab Alizadeh, Brianna M. Otero, Afrooz Jalilzadeh

Abstract: This paper is focused on a stochastic quasi-variational inequality (SQVI) problem with a continuous and strongly-monotone mapping over a closed and convex set where the projection onto the constraint set may not be easy to compute. We present an inexact variance reduced stochastic scheme to solve SQVI problems and analyzed its convergence rate and oracle complexity. A linear rate of convergence is… ▽ More This paper is focused on a stochastic quasi-variational inequality (SQVI) problem with a continuous and strongly-monotone mapping over a closed and convex set where the projection onto the constraint set may not be easy to compute. We present an inexact variance reduced stochastic scheme to solve SQVI problems and analyzed its convergence rate and oracle complexity. A linear rate of convergence is obtained by progressively increasing sample-size and approximating the projection operator. Moreover, we show how a competition among blood donation organizations can be modeled as an SQVI and we provide some preliminary simulation results to validate our findings. △ Less

Submitted 31 August, 2022; originally announced September 2022.

Stochastic Approximation for Estimating the Price of Stability in Stochastic Nash Games

Authors: Afrooz Jalilzadeh, Farzad Yousefian, Mohammadjavad Ebrahimi

Abstract: The goal in this paper is to approximate the Price of Stability (PoS) in stochastic Nash games using stochastic approximation (SA) schemes. PoS is amongst the most popular metrics in game theory and provides an avenue for estimating the efficiency of Nash games. In particular, knowing the value of PoS can help with designing efficient networked systems, including transportation networks and power… ▽ More The goal in this paper is to approximate the Price of Stability (PoS) in stochastic Nash games using stochastic approximation (SA) schemes. PoS is amongst the most popular metrics in game theory and provides an avenue for estimating the efficiency of Nash games. In particular, knowing the value of PoS can help with designing efficient networked systems, including transportation networks and power market mechanisms. Motivated by the lack of efficient methods for computing the PoS, first we consider stochastic optimization problems with a nonsmooth and merely convex objective function and a merely monotone stochastic variational inequality (SVI) constraint. This problem appears in the numerator of the PoS ratio. We develop a randomized block-coordinate stochastic extra-(sub)gradient method where we employ a novel iterative penalization scheme to account for the mapping of the SVI in each of the two gradient updates of the algorithm. We obtain an iteration complexity of the order $ε^{-4}$ that appears to be best known result for this class of constrained stochastic optimization problems, where $ε$ denotes an arbitrary bound on suitably defined infeasibility and suboptimality metrics. Second, we develop an SA-based scheme for approximating the PoS and derive lower and upper bounds on the approximation error. To validate the theoretical findings, we provide preliminary simulation results on a networked stochastic Nash Cournot competition. △ Less

Submitted 28 October, 2023; v1 submitted 2 March, 2022; originally announced March 2022.

Inexact-Proximal Accelerated Gradient Method for Stochastic Nonconvex Constrained Optimization Problems

Authors: Morteza Boroun, Afrooz Jalilzadeh

Abstract: Stochastic nonconvex optimization problems with nonlinear constraints have a broad range of applications in intelligent transportation, cyber-security, and smart grids. In this paper, first, we propose an inexact-proximal accelerated gradient method to solve a nonconvex stochastic composite optimization problem where the objective is the sum of smooth and nonsmooth functions, the constraint functi… ▽ More Stochastic nonconvex optimization problems with nonlinear constraints have a broad range of applications in intelligent transportation, cyber-security, and smart grids. In this paper, first, we propose an inexact-proximal accelerated gradient method to solve a nonconvex stochastic composite optimization problem where the objective is the sum of smooth and nonsmooth functions, the constraint functions are assumed to be deterministic and the solution to the proximal map of the nonsmooth part is calculated inexactly at each iteration. We demonstrate an asymptotic sublinear rate of convergence for stochastic settings using increasing sample-size considering the error in the proximal operator diminishes at an appropriate rate. Then we customize the proposed method for solving stochastic nonconvex optimization problems with nonlinear constraints and demonstrate a convergence rate guarantee. Numerical results show the effectiveness of the proposed algorithm. △ Less

Submitted 7 July, 2021; v1 submitted 15 April, 2021; originally announced April 2021.

A Stochastic Variance-reduced Accelerated Primal-dual Method for Finite-sum Saddle-point Problems

Authors: Erfan Yazdandoost Hamedani, Afrooz Jalilzadeh

Abstract: In this paper, we propose a variance-reduced primal-dual algorithm with Bregman distance for solving convex-concave saddle-point problems with finite-sum structure and nonbilinear coupling function. This type of problems typically arises in machine learning and game theory. Based on some standard assumptions, the algorithm is proved to converge with oracle complexity of… ▽ More In this paper, we propose a variance-reduced primal-dual algorithm with Bregman distance for solving convex-concave saddle-point problems with finite-sum structure and nonbilinear coupling function. This type of problems typically arises in machine learning and game theory. Based on some standard assumptions, the algorithm is proved to converge with oracle complexity of $O\left(\frac{\sqrt n}ε\right)$ and $O\left(\frac{n}{\sqrt ε}+\frac{1}{ε^{1.5}}\right)$ using constant and non-constant parameters, respectively where $n$ is the number of function components. Compared with existing methods, our framework yields a significant improvement over the number of required primal-dual gradient samples to achieve $ε$-accuracy of the primal-dual gap. We tested our method for solving a distributionally robust optimization problem to show the effectiveness of the algorithm. △ Less

Submitted 31 May, 2021; v1 submitted 24 December, 2020; originally announced December 2020.

Primal-Dual Incremental Gradient Method for Nonsmooth and Convex Optimization Problems

Authors: Afrooz Jalilzadeh

Abstract: In this paper, we consider a nonsmooth convex finite-sum problem with a conic constraint. To overcome the challenge of projecting onto the constraint set and computing the full (sub)gradient, we introduce a primal-dual incremental gradient scheme where only a component function and two constraints are used to update each primal-dual sub-iteration in a cyclic order. We demonstrate an asymptotic sub… ▽ More In this paper, we consider a nonsmooth convex finite-sum problem with a conic constraint. To overcome the challenge of projecting onto the constraint set and computing the full (sub)gradient, we introduce a primal-dual incremental gradient scheme where only a component function and two constraints are used to update each primal-dual sub-iteration in a cyclic order. We demonstrate an asymptotic sublinear rate of convergence in terms of suboptimality and infeasibility which is an improvement over the state-of-the-art incremental gradient schemes in this setting. Numerical results suggest that the proposed scheme compares well with competitive methods. △ Less

Submitted 7 May, 2021; v1 submitted 3 November, 2020; originally announced November 2020.

A Proximal-Point Algorithm with Variable Sample-sizes (PPAWSS) for Monotone Stochastic Variational Inequality Problems

Authors: Afrooz Jalilzadeh, Uday V. Shanbhag

Abstract: We consider a stochastic variational inequality (SVI) problem with a continuous and monotone mapping over a closed and convex set. In strongly monotone regimes, we present a variable sample-size averaging scheme (VS-Ave) that achieves a linear rate with an optimal oracle complexity. In addition, the iteration complexity is shown to display a muted dependence on the condition number compared with s… ▽ More We consider a stochastic variational inequality (SVI) problem with a continuous and monotone mapping over a closed and convex set. In strongly monotone regimes, we present a variable sample-size averaging scheme (VS-Ave) that achieves a linear rate with an optimal oracle complexity. In addition, the iteration complexity is shown to display a muted dependence on the condition number compared with standard variance-reduced projection schemes. To contend with merely monotone maps, we develop amongst the first proximal-point algorithms with variable sample-sizes (PPAWSS), where increasingly accurate solutions of strongly monotone SVIs are obtained via (VS-Ave) at every step. This allows for achieving a sublinear convergence rate that matches that obtained for deterministic monotone VIs. Preliminary numerical evidence suggests that the schemes compares well with competing schemes. △ Less

Submitted 29 September, 2019; originally announced September 2019.

A Randomized Block-Coordinate Primal-Dual Method for Large-scale Stochastic Saddle Point Problems

Authors: Erfan Yazdandoost Hamedani, Afrooz Jalilzadeh, Necdet Serhat Aybat

Abstract: We consider (stochastic) convex-concave saddle point (SP) problems with high-dimensional decision variables, arising in various machine learning problems. To contend with the challenges in computing full gradients, we employ a randomized block-coordinate primal-dual scheme in which randomly selected primal and dual blocks of variables are updated. We consider both deterministic and stochastic sett… ▽ More We consider (stochastic) convex-concave saddle point (SP) problems with high-dimensional decision variables, arising in various machine learning problems. To contend with the challenges in computing full gradients, we employ a randomized block-coordinate primal-dual scheme in which randomly selected primal and dual blocks of variables are updated. We consider both deterministic and stochastic settings, where deterministic partial gradients and their randomly sampled estimates are used, respectively, at each iteration. We investigate the convergence of the proposed method under different blocking strategies and provide the corresponding complexity results. While the best-known complexity result for deterministic primal-dual methods using full gradients is $\mathcal O(\max\{M,N\}^2/\varepsilon)$ where $M$ and $N$ denote the number of primal and dual blocks, respectively, we show that our proposed randomized block-coordinate method can achieve an improved convergence rate of $\mathcal O(MN/\varepsilon)$. Moreover, for the stochastic setting where a mini-batch sample gradient is utilized, we show an optimal oracle complexity of $\tilde{\mathcal{O}}(M^2N^2/\varepsilon^2)$ through acceleration. Finally, almost sure convergence of the iterate sequence to a saddle point is established. △ Less

Submitted 10 October, 2024; v1 submitted 8 July, 2019; originally announced July 2019.

Comments: This is a major update. Complexity results are established under stochastic first-order oracle setting rather than the finite-sum form considered in earlier versions. All the theorems and their proofs are revised. We discuss when blocking both the primal and dual vector leads to better complexity bounds than the complexity without using blocks or with blocking either the primal or dual alone

Randomized Primal-Dual Methods with Adaptive Step Sizes

Authors: E. Yazdandoost Hamedani, A. Jalilzadeh, N. S. Aybat

Abstract: In this paper we propose a class of randomized primal-dual methods to contend with large-scale saddle point problems defined by a convex-concave function $\mathcal{L}(\mathbf{x},y)\triangleq\sum_{i=1}^m f_i(x_i)+Φ(\mathbf{x},y)-h(y)$. We analyze the convergence rate of the proposed method under mere convexity and strong convexity assumptions of $\mathcal{L}$ in $\mathbf{x}$-variable. In particular… ▽ More In this paper we propose a class of randomized primal-dual methods to contend with large-scale saddle point problems defined by a convex-concave function $\mathcal{L}(\mathbf{x},y)\triangleq\sum_{i=1}^m f_i(x_i)+Φ(\mathbf{x},y)-h(y)$. We analyze the convergence rate of the proposed method under mere convexity and strong convexity assumptions of $\mathcal{L}$ in $\mathbf{x}$-variable. In particular, assuming $\nabla_yΦ(\cdot,\cdot)$ is Lipschitz and $\nabla_\mathbf{x}Φ(\cdot,y)$ is coordinate-wise Lipschitz for any fixed $y$, the ergodic sequence generated by the algorithm achieves the convergence rate of $\mathcal{O}(M/k)$ in the expected primal-dual gap. Furthermore, assuming that $\mathcal{L}(\cdot,y)$ is strongly convex for any $y$, and that $Φ(\mathbf{x},\cdot)$ is affine for any $\mathbf{x}$, the scheme enjoys a faster rate of $\mathcal{O}(M/k^2)$ in terms of primal solution suboptimality. We implemented the proposed algorithmic framework to solve kernel matrix learning problem, and tested it against other state-of-the-art first-order methods △ Less

Submitted 16 March, 2023; v1 submitted 11 June, 2018; originally announced June 2018.

Comments: Accepted at AISTATS 2023

A Variable Sample-size Stochastic Quasi-Newton Method for Smooth and Nonsmooth Stochastic Convex Optimization

Authors: Afrooz Jalilzadeh, Angelia Nedich, Uday V. Shanbhag, Farzad Yousefian

Abstract: Classical theory for quasi-Newton schemes has focused on smooth deterministic unconstrained optimization while recent forays into stochastic convex optimization have largely resided in smooth, unconstrained, and strongly convex regimes. Naturally, there is a compelling need to address nonsmoothness, the lack of strong convexity, and the presence of constraints. Accordingly, this paper presents a q… ▽ More Classical theory for quasi-Newton schemes has focused on smooth deterministic unconstrained optimization while recent forays into stochastic convex optimization have largely resided in smooth, unconstrained, and strongly convex regimes. Naturally, there is a compelling need to address nonsmoothness, the lack of strong convexity, and the presence of constraints. Accordingly, this paper presents a quasi-Newton framework that can process merely convex and possibly nonsmooth (but smoothable) stochastic convex problems. We propose a framework that combines iterative smoothing and regularization with a variance-reduced scheme reliant on using increasing sample-sizes of gradients. We make the following contributions. (i) We develop a regularized and smoothed variable sample-size BFGS update (rsL-BFGS) that generates a sequence of Hessian approximations and can accommodate nonsmooth convex objectives by utilizing iterative regularization and smoothing. (ii) In strongly convex regimes with state-dependent noise, the proposed variable sample-size stochastic quasi-Newton scheme admits a non-asymptotic linear rate of convergence while the oracle complexity of computing an $ε$-solution is $\mathcal{O}(κ^{m+1}/ε)$ where $κ$ is the condition number and $m\geq 1$. In nonsmooth (but smoothable) regimes, using Moreau smoothing retains the linear convergence rate. To contend with the possible unavailability of Lipschitzian and strong convexity parameters, we also provide sublinear rates; (iii) In merely convex but smooth settings, the regularized VS-SQN scheme rVS-SQN displays a rate of $\mathcal{O}(1/k^{(1-\varepsilon)})$. When the smoothness requirements are weakened, the rate for the regularized and smoothed VS-SQN scheme worsens to $\mathcal{O}(k^{-1/3})$. Such statements allow for a state-dependent noise assumption under a quadratic growth property. △ Less

Submitted 2 November, 2020; v1 submitted 15 April, 2018; originally announced April 2018.

Smoothed Variable Sample-size Accelerated Proximal Methods for Nonsmooth Stochastic Convex Programs

Authors: Afrooz Jalilzadeh, Uday V. Shanbhag, Jose H. Blanchet, Peter W. Glynn

Abstract: We consider minimizing $f(x) = \mathbb{E}[f(x,ω)]$ when $f(x,ω)$ is possibly nonsmooth and either strongly convex or convex in $x$. (I) Strongly convex. When $f(x,ω)$ is $μ-$strongly convex in $x$, we propose a variable sample-size accelerated proximal scheme (VS-APM) and apply it on $f_η(x)$, the ($η$-)Moreau smoothed variant of $\mathbb{E}[f(x,ω)]$; we term such a scheme as (m-VS-APM). We consid… ▽ More We consider minimizing $f(x) = \mathbb{E}[f(x,ω)]$ when $f(x,ω)$ is possibly nonsmooth and either strongly convex or convex in $x$. (I) Strongly convex. When $f(x,ω)$ is $μ-$strongly convex in $x$, we propose a variable sample-size accelerated proximal scheme (VS-APM) and apply it on $f_η(x)$, the ($η$-)Moreau smoothed variant of $\mathbb{E}[f(x,ω)]$; we term such a scheme as (m-VS-APM). We consider three settings. (a) Bounded domains. In this setting, VS-APM displays linear convergence in inexact gradient steps, each of which requires utilizing an inner (SSG) scheme. Specifically, mVS-APM achieves an optimal oracle complexity in SSG steps; (b) Unbounded domains. In this regime, under a weaker assumption of suitable state-dependent bounds on subgradients, an unaccelerated variant mVS-PM is linearly convergent; (c) Smooth ill-conditioned $f$. When $f$ is $L$-smooth and $κ= L/μ\ggg 1$, we employ mVS-APM where increasingly accurate gradients $\nabla_x f_η(x)$ are obtained by VS-APM. Notably, mVS-APM displays linear convergence and near-optimal complexity in inner proximal evaluations (upto a log factor) compared to VS-APM. But, unlike a direct application of VS-APM, this scheme is characterized by larger steplengths and better empirical behavior; (II) Convex. When $f(x,ω)$ is merely convex but smoothable, by suitable choices of the smoothing, steplength, and batch-size sequences, smoothed VS-APM (or sVS-APM) produces sequences for which expected sub-optimality diminishes at the rate of $\mathcal{O}(1/k)$ with an optimal oracle complexity of $\mathcal{O}(1/ε^2)$. Finally, sVS-APM and VS-APM produce sequences that converge almost surely to a solution of the original problem. △ Less

Submitted 6 October, 2022; v1 submitted 1 March, 2018; originally announced March 2018.

Comments: Stochastic Systems (2022)

Probability Maximization via Minkowski Functionals: Convex Representations and Tractable Resolution

Authors: Ibrahim E. Bardakci, Afrooz Jalilzadeh, Constantino Lagoa, Uday V. Shanbhag

Abstract: In this paper, we consider the maximization of a probability $\mathbb{P}\{ ζ\mid ζ\in \mathbf{K}(\mathbf x)\}$ over a closed and convex set $\mathcal X$, a special case of the chance-constrained optimization problem. We define $\mathbf{K}(\mathbf x)$ as $\mathbf{K}(\mathbf x) \triangleq \{ ζ\in \mathcal{K} \mid c(\mathbf{x},ζ) \geq 0 \}$ where $ζ$ is uniformly distributed on a convex and compact s… ▽ More In this paper, we consider the maximization of a probability $\mathbb{P}\{ ζ\mid ζ\in \mathbf{K}(\mathbf x)\}$ over a closed and convex set $\mathcal X$, a special case of the chance-constrained optimization problem. We define $\mathbf{K}(\mathbf x)$ as $\mathbf{K}(\mathbf x) \triangleq \{ ζ\in \mathcal{K} \mid c(\mathbf{x},ζ) \geq 0 \}$ where $ζ$ is uniformly distributed on a convex and compact set $\mathcal{K}$ and $c(\mathbf{x},ζ)$ is defined as either {$c(\mathbf{x},ζ) \triangleq 1-|ζ^T\mathbf{x}|^m$, $m\geq 0$} (Setting A) or $c(\mathbf{x},ζ) \triangleq T\mathbf{x} -ζ$ (Setting B). We show that in either setting, $\mathbb{P}\{ ζ\mid ζ\in \mathbf{K(x)}\}$ can be expressed as the expectation of a suitably defined function $F(\mathbf{x},ξ)$ with respect to an appropriately defined Gaussian density (or its variant), i.e. $\mathbb{E}_{\tilde p} [F(\mathbf x,ξ)]$. We then develop a convex representation of the original problem requiring the minimization of ${g(\mathbb{E}[F(\mathbf{x},ξ)])}$ over $\mathcal X$ where $g$ is an appropriately defined smooth convex function. Traditional stochastic approximation schemes cannot contend with the minimization of ${g(\mathbb{E}[F(\cdot,ξ)])}$ over $\mathcal X$, since conditionally unbiased sampled gradients are unavailable. We then develop a regularized variance-reduced stochastic approximation (r-VRSA) scheme that obviates the need for such unbiasedness by combining iterative regularization with variance-reduction. Notably, (r-VRSA) is characterized by both almost-sure convergence guarantees, a convergence rate of $\mathcal{O}(1/k^{1/2-a})$ in expected sub-optimality where $a > 0$, and a sample complexity of $\mathcal{O}(1/ε^{6+δ})$ where $δ> 0$. △ Less

Submitted 10 March, 2022; v1 submitted 26 February, 2018; originally announced February 2018.

Inverse Quadratic Transportation Problem

Authors: Afrooz Jalilzadeh, Erfan Yazdandoost Hamedani

Abstract: Many research has been conducted about quadratic programming and inverse optimization. In this paper we present the combination aspect of these subjects, applying on transportation problem. First, we obtain the inverse form of quadratic tranportation problem under $L_1$ norm by using duality as well as introducing the optimal value. Then, we do the same process for inverse quadratic transportation… ▽ More Many research has been conducted about quadratic programming and inverse optimization. In this paper we present the combination aspect of these subjects, applying on transportation problem. First, we obtain the inverse form of quadratic tranportation problem under $L_1$ norm by using duality as well as introducing the optimal value. Then, we do the same process for inverse quadratic transportation problem (IQTP) under $L_\infty$ norm. △ Less

Submitted 24 September, 2014; v1 submitted 21 September, 2014; originally announced September 2014.