Showing 1–27 of 27 results for author: Phan, H M

Projected proximal gradient trust-region algorithm for nonsmooth optimization

Authors: Minh N. Dao, Hung M. Phan, Lindon Roberts

Abstract: We consider trust-region methods for solving optimization problems where the objective is the sum of a smooth, nonconvex function and a nonsmooth, convex regularizer. We extend the global convergence theory of such methods to include worst-case complexity bounds in the case of unbounded model Hessian growth, and introduce a new, simple nonsmooth trust-region subproblem solver based on combining se… ▽ More We consider trust-region methods for solving optimization problems where the objective is the sum of a smooth, nonconvex function and a nonsmooth, convex regularizer. We extend the global convergence theory of such methods to include worst-case complexity bounds in the case of unbounded model Hessian growth, and introduce a new, simple nonsmooth trust-region subproblem solver based on combining several iterations of proximal gradient descent with a single projection into the trust region, which meets the sufficient descent requirements for algorithm convergence and has promising numerical results. △ Less

Submitted 8 January, 2025; originally announced January 2025.

Stochastic Variance-Reduced Majorization-Minimization Algorithms

Authors: Duy-Nhat Phan, Sedi Bartz, Nilabja Guha, Hung M. Phan

Abstract: We study a class of nonconvex nonsmooth optimization problems in which the objective is a sum of two functions: One function is the average of a large number of differentiable functions, while the other function is proper, lower semicontinuous and has a surrogate function that satisfies standard assumptions. Such problems arise in machine learning and regularized empirical risk minimization applic… ▽ More We study a class of nonconvex nonsmooth optimization problems in which the objective is a sum of two functions: One function is the average of a large number of differentiable functions, while the other function is proper, lower semicontinuous and has a surrogate function that satisfies standard assumptions. Such problems arise in machine learning and regularized empirical risk minimization applications. However, nonconvexity and the large-sum structure are challenging for the design of new algorithms. Consequently, effective algorithms for such scenarios are scarce. We introduce and study three stochastic variance-reduced majorization-minimization (MM) algorithms, combining the general MM principle with new variance-reduced techniques. We provide almost surely subsequential convergence of the generated sequence to a stationary point. We further show that our algorithms possess the best-known complexity bounds in terms of gradient evaluations. We demonstrate the effectiveness of our algorithms on sparse binary classification problems, sparse multi-class logistic regressions, and neural networks by employing several widely-used and publicly available data sets. △ Less

Submitted 11 May, 2023; originally announced May 2023.

MSC Class: 90C26; 65K05

An adaptive splitting algorithm for the sum of three operators

Authors: Minh N. Dao, Hung M. Phan

Abstract: Splitting algorithms for finding a zero of sum of operators often involve multiple steps which are referred to as forward or backward steps. Forward steps are the explicit use of the operators and backward steps involve the operators implicitly via their resolvents. In this paper, we study an adaptive splitting algorithm for finding a zero of the sum of three operators. We assume that two of the o… ▽ More Splitting algorithms for finding a zero of sum of operators often involve multiple steps which are referred to as forward or backward steps. Forward steps are the explicit use of the operators and backward steps involve the operators implicitly via their resolvents. In this paper, we study an adaptive splitting algorithm for finding a zero of the sum of three operators. We assume that two of the operators are generalized monotone and their resolvents are computable, while the other operator is cocoercive but its resolvent is missing or costly to compute. Our splitting algorithm adapts new parameters to the generalized monotonicity of the operators and, at the same time, combines appropriate forward and backward steps to guarantee convergence to a solution of the problem. △ Less

Submitted 12 April, 2021; originally announced April 2021.

An Adaptive Alternating Direction Method of Multipliers

Authors: Sedi Bartz, Rubén Campoy, Hung M. Phan

Abstract: The alternating direction method of multipliers (ADMM) is a powerful splitting algorithm for linearly constrained convex optimization problems. In view of its popularity and applicability, a growing attention is drawn towards the ADMM in nonconvex settings. Recent studies of minimization problems for noncovex functions include various combinations of assumptions on the objective function including… ▽ More The alternating direction method of multipliers (ADMM) is a powerful splitting algorithm for linearly constrained convex optimization problems. In view of its popularity and applicability, a growing attention is drawn towards the ADMM in nonconvex settings. Recent studies of minimization problems for noncovex functions include various combinations of assumptions on the objective function including, in particular, a Lipschitz gradient assumption. We consider the case where the objective is the sum of a strongly convex function and a weakly convex function. To this end we present and study an adaptive version of the ADMM which incorporates generalized notions of convexity and varying penalty parameters adapted to the convexity constants of the functions. We prove convergence of the scheme under natural assumptions. To this end we employ the recent adaptive Douglas--Rachford algorithm by revisiting the well known duality relation between the classical ADMM and the Douglas--Rachford splitting algorithm, generalizing this connection to our setting. We illustrate our approach by relating and comparing to alternatives, and by numerical experiments on a signal denoising problem. △ Less

Submitted 17 August, 2022; v1 submitted 12 March, 2021; originally announced March 2021.

MSC Class: 47H05; 47N10; 47J25; 49M27; 65K15

Demiclosedness principles for generalized nonexpansive mappings

Authors: Sedi Bartz, Rubén Campoy, Hung M. Phan

Abstract: Demiclosedness principles are powerful tools in the study of convergence of iterative methods. For instance, a multi-operator demiclosedness principle for firmly nonexpansive mappings is useful in obtaining simple and transparent arguments for the weak convergence of the shadow sequence generated by the Douglas-Rachford algorithm. We provide extensions of this principle which are compatible with t… ▽ More Demiclosedness principles are powerful tools in the study of convergence of iterative methods. For instance, a multi-operator demiclosedness principle for firmly nonexpansive mappings is useful in obtaining simple and transparent arguments for the weak convergence of the shadow sequence generated by the Douglas-Rachford algorithm. We provide extensions of this principle which are compatible with the framework of more general families of mappings such as cocoercive and conically averaged mappings. As an application, we derive the weak convergence of the shadow sequence generated by the adaptive Douglas-Rachford algorithm. △ Less

Submitted 24 August, 2020; v1 submitted 28 April, 2020; originally announced April 2020.

MSC Class: 47H05; 47J25; 49M27

Conical averagedness and convergence analysis of fixed point algorithms

Authors: Sedi Bartz, Minh N. Dao, Hung M. Phan

Abstract: We study a conical extension of averaged nonexpansive operators and the role it plays in convergence analysis of fixed point algorithms. Various properties of conically averaged operators are systematically investigated, in particular, the stability under relaxations, convex combinations and compositions. We derive conical averagedness properties of resolvents of generalized monotone operators. Th… ▽ More We study a conical extension of averaged nonexpansive operators and the role it plays in convergence analysis of fixed point algorithms. Various properties of conically averaged operators are systematically investigated, in particular, the stability under relaxations, convex combinations and compositions. We derive conical averagedness properties of resolvents of generalized monotone operators. These properties are then utilized in order to analyze the convergence of the proximal point algorithm, the forward-backward algorithm, and the adaptive Douglas-Rachford algorithm. Our study unifies, improves and casts new light on recent studies of these topics. △ Less

Submitted 30 November, 2020; v1 submitted 30 October, 2019; originally announced October 2019.

MSC Class: Primary: 47H10; 49M27; Secondary: 65K05; 65K10

Multi-marginal maximal monotonicity and convex analysis

Authors: Sedi Bartz, Heinz H. Bauschke, Hung M. Phan, Xianfu Wang

Abstract: Monotonicity and convex analysis arise naturally in the framework of multi-marginal optimal transport theory. However, a comprehensive multi-marginal monotonicity and convex analysis theory is still missing. To this end we study extensions of classical monotone operator theory and convex analysis into the multi-marginal setting. We characterize multi-marginal c-monotonicity in terms of classical m… ▽ More Monotonicity and convex analysis arise naturally in the framework of multi-marginal optimal transport theory. However, a comprehensive multi-marginal monotonicity and convex analysis theory is still missing. To this end we study extensions of classical monotone operator theory and convex analysis into the multi-marginal setting. We characterize multi-marginal c-monotonicity in terms of classical monotonicity and firmly nonexpansive mappings. We provide Minty type, continuity and conjugacy criteria for multi-marginal maximal monotonicity. We extend the partition of the identity into a sum of firmly nonexpansive mappings and Moreau's decomposition of the quadratic function into envelopes and proximal mappings into the multi-marginal settings. We illustrate our discussion with examples and provide applications for the determination of multi-marginal maximal monotonicity and multi-marginal conjugacy. We also point out several open questions. △ Less

Submitted 18 September, 2019; v1 submitted 11 January, 2019; originally announced January 2019.

MSC Class: Primary 47H05; 26B25; Secondary 49N15; 49K30; 52A01; 91B68

Optimization of triangular networks with spatial constraints

Authors: Valentin R. Koch, Hung M. Phan

Abstract: A common representation of a three dimensional object in computer applications, such as graphics and design, is in the form of a triangular mesh. In many instances, individual or groups of triangles in such representation need to satisfy spatial constraints that are imposed either by observation from the real world, or by concrete design specifications of the object. As these problems tend to be o… ▽ More A common representation of a three dimensional object in computer applications, such as graphics and design, is in the form of a triangular mesh. In many instances, individual or groups of triangles in such representation need to satisfy spatial constraints that are imposed either by observation from the real world, or by concrete design specifications of the object. As these problems tend to be of large scale, choosing a mathematical optimization approach can be particularly challenging. In this paper, we model various geometric constraints as convex sets in Euclidean spaces, and find the corresponding projections in closed forms. We also present an interesting idea to successfully maneuver around some important nonconvex constraints while still preserving the intrinsic nature of the original design problem. We then use these constructions in modern first-order splitting methods to find optimal solutions. △ Less

Submitted 4 April, 2019; v1 submitted 12 November, 2018; originally announced November 2018.

MSC Class: 26B25; 65D17 (Primary); 49M27; 52A41; 90C25 (Secondary)

Computing the resolvent of the sum of operators with application to best approximation problems

Authors: Minh N. Dao, Hung M. Phan

Abstract: We propose a flexible approach for computing the resolvent of the sum of weakly monotone operators in real Hilbert spaces. This relies on splitting methods where strong convergence is guaranteed. We also prove linear convergence under Lipschitz continuity assumption. The approach is then applied to computing the proximity operator of the sum of weakly convex functions, and particularly to finding… ▽ More We propose a flexible approach for computing the resolvent of the sum of weakly monotone operators in real Hilbert spaces. This relies on splitting methods where strong convergence is guaranteed. We also prove linear convergence under Lipschitz continuity assumption. The approach is then applied to computing the proximity operator of the sum of weakly convex functions, and particularly to finding the best approximation to the intersection of convex sets. △ Less

Submitted 11 September, 2018; originally announced September 2018.

MSC Class: Primary: 47H05; 49M27; Secondary: 47H10; 65K05; 65K10

Adaptive Douglas-Rachford splitting algorithm for the sum of two operators

Authors: Minh N. Dao, Hung M. Phan

Abstract: The Douglas-Rachford algorithm is a classical and powerful splitting method for minimizing the sum of two convex functions and, more generally, finding a zero of the sum of two maximally monotone operators. Although this algorithm is well understood when the involved operators are monotone or strongly monotone, the convergence theory for weakly monotone settings is far from being complete. In this… ▽ More The Douglas-Rachford algorithm is a classical and powerful splitting method for minimizing the sum of two convex functions and, more generally, finding a zero of the sum of two maximally monotone operators. Although this algorithm is well understood when the involved operators are monotone or strongly monotone, the convergence theory for weakly monotone settings is far from being complete. In this paper, we propose an adaptive Douglas-Rachford splitting algorithm for the sum of two operators, one of which is strongly monotone while the other one is weakly monotone. With appropriately chosen parameters, the algorithm converges globally to a fixed point from which we derive a solution of the problem. When one operator is Lipschitz continuous, we prove global linear convergence, which sharpens recent known results. △ Less

Submitted 9 October, 2019; v1 submitted 3 September, 2018; originally announced September 2018.

Comments: Final version for SIAM Journal on Optimization

MSC Class: Primary: 47H10; 49M27; Secondary: 41A25; 65K05; 65K10

Journal ref: SIAM Journal on Optimization, 29(4):2697--2724, 2019

Linear convergence of the generalized Douglas-Rachford algorithm for feasibility problems

Authors: Minh N. Dao, Hung M. Phan

Abstract: In this paper, we study the generalized Douglas-Rachford algorithm and its cyclic variants which include many projection-type methods such as the classical Douglas-Rachford algorithm and the alternating projection algorithm. Specifically, we establish several local linear convergence results for the algorithm in solving feasibility problems with finitely many closed possibly nonconvex sets under d… ▽ More In this paper, we study the generalized Douglas-Rachford algorithm and its cyclic variants which include many projection-type methods such as the classical Douglas-Rachford algorithm and the alternating projection algorithm. Specifically, we establish several local linear convergence results for the algorithm in solving feasibility problems with finitely many closed possibly nonconvex sets under different assumptions. Our findings not only relax some regularity conditions but also improve linear convergence rates in the literature. In the presence of convexity, the linear convergence is global. △ Less

Submitted 16 April, 2018; v1 submitted 26 October, 2017; originally announced October 2017.

MSC Class: 47H10; 49M27 (Primary); 41A25; 65K05; 65K10; 90C26 (Secondary)

Journal ref: Journal of Global Optimization, 72(3):443--474, 2018

Linear Convergence of Projection Algorithms

Authors: Minh N. Dao, Hung M. Phan

Abstract: Projection algorithms are well known for their simplicity and flexibility in solving feasibility problems. They are particularly important in practice due to minimal requirements for software implementation and maintenance. In this work, we study linear convergence of several projection algorithms for systems of finitely many closed sets. The results complement contemporary research on the same to… ▽ More Projection algorithms are well known for their simplicity and flexibility in solving feasibility problems. They are particularly important in practice due to minimal requirements for software implementation and maintenance. In this work, we study linear convergence of several projection algorithms for systems of finitely many closed sets. The results complement contemporary research on the same topic. △ Less

Submitted 25 October, 2017; v1 submitted 1 September, 2016; originally announced September 2016.

MSC Class: 49M27; 65K10 (Primary) 47H09; 49J52; 49M37; 65K05; 90C26 (Secondary)

Journal ref: Mathematics of Operations Research, 44(2):715--738, 2019

On Slater's condition and finite convergence of the Douglas-Rachford algorithm

Authors: Heinz H. Bauschke, Minh N. Dao, Dominikus Noll, Hung M. Phan

Abstract: The Douglas-Rachford algorithm is a classical and very successful method for solving optimization and feasibility problems. In this paper, we provide novel conditions sufficient for finite convergence in the context of convex feasibility problems. Our analysis builds upon, and considerably extends, pioneering work by Spingarn. Specifically, we obtain finite convergence in the presence of Slater's… ▽ More The Douglas-Rachford algorithm is a classical and very successful method for solving optimization and feasibility problems. In this paper, we provide novel conditions sufficient for finite convergence in the context of convex feasibility problems. Our analysis builds upon, and considerably extends, pioneering work by Spingarn. Specifically, we obtain finite convergence in the presence of Slater's condition in the affine-polyhedral and in a hyperplanar-epigraphical case. Various examples illustrate our results. Numerical experiments demonstrate the competitiveness of the Douglas-Rachford algorithm for solving linear equations with a positivity constraint when compared to the method of alternating projections and the method of reflection-projection. △ Less

Submitted 27 April, 2015; originally announced April 2015.

MSC Class: Primary 47H09; 90C25; Secondary 47H05; 49M27; 65F10; 65K05; 65K10

Journal ref: Journal of Global Optimization, 65(2):329--349, 2016

Proximal point algorithm, Douglas-Rachford algorithm and alternating projections: a case study

Authors: Heinz H. Bauschke, Minh N. Dao, Dominikus Noll, Hung M. Phan

Abstract: Many iterative methods for solving optimization or feasibility problems have been invented, and often convergence of the iterates to some solution is proven. Under favourable conditions, one might have additional bounds on the distance of the iterate to the solution leading thus to worst case estimates, i.e., how fast the algorithm must converge. Exact convergence estimates are typically hard to… ▽ More Many iterative methods for solving optimization or feasibility problems have been invented, and often convergence of the iterates to some solution is proven. Under favourable conditions, one might have additional bounds on the distance of the iterate to the solution leading thus to worst case estimates, i.e., how fast the algorithm must converge. Exact convergence estimates are typically hard to come by. In this paper, we consider the complementary problem of finding best case estimates, i.e., how slow the algorithm has to converge, and we also study exact asymptotic rates of convergence. Our investigation focuses on convex feasibility in the Euclidean plane, where one set is the real axis while the other is the epigraph of a convex function. This case study allows us to obtain various convergence rate results. We focus on the popular method of alternating projections and the Douglas-Rachford algorithm. These methods are connected to the proximal point algorithm which is also discussed. Our findings suggest that the Douglas-Rachford algorithm outperforms the method of alternating projections in the absence of constraint qualifications. Various examples illustrate the theory. △ Less

Submitted 26 January, 2015; originally announced January 2015.

MSC Class: Primary 65K05; Secondary 65K10; 90C25

Journal ref: Journal of Convex Analysis, 23(1):237--261, 2016

Conditional extragradient algorithms for solving variational inequalities

Authors: J. Y. Bello Cruz, R. Díaz Millán, Hung M. Phan

Abstract: In this paper, we generalize the classical extragradient algorithm for solving variational inequality problems by utilizing nonzero normal vectors of the feasible set. In particular, conceptual algorithms are proposed with two different linesearchs. We then establish convergence results for these algorithms under mild assumptions. Our study suggests that nonzero normal vectors may significantly im… ▽ More In this paper, we generalize the classical extragradient algorithm for solving variational inequality problems by utilizing nonzero normal vectors of the feasible set. In particular, conceptual algorithms are proposed with two different linesearchs. We then establish convergence results for these algorithms under mild assumptions. Our study suggests that nonzero normal vectors may significantly improve convergence if chosen appropriately. △ Less

Submitted 22 February, 2019; v1 submitted 16 November, 2014; originally announced November 2014.

MSC Class: Primary: 58E35; Secondary: 49J40; 65K15

Stadium norm and Douglas-Rachford splitting: a new approach to road design optimization

Authors: Heinz H. Bauschke, Valentin R. Koch, Hung M. Phan

Abstract: The basic optimization problem of road design is quite challenging due to a objective function that is the sum of nonsmooth functions and the presence of set constraints. In this paper, we model and solve this problem by employing the Douglas-Rachford splitting algorithm. This requires a careful study of new proximity operators related to minimizing area and to the stadium norm. We compare our alg… ▽ More The basic optimization problem of road design is quite challenging due to a objective function that is the sum of nonsmooth functions and the presence of set constraints. In this paper, we model and solve this problem by employing the Douglas-Rachford splitting algorithm. This requires a careful study of new proximity operators related to minimizing area and to the stadium norm. We compare our algorithm to a state-of-the-art projection algorithm. Our numerical results illustrate the potential of this algorithm to significantly reduce cost in road design. △ Less

Submitted 29 September, 2014; originally announced September 2014.

MSC Class: Primary 65K05; 90C25; Secondary 41A65; 49M27; 49M37; 52A21

On the convexity of piecewise-defined functions

Authors: Heinz H. Bauschke, Yves Lucet, Hung M. Phan

Abstract: Functions that are piecewise defined are a common sight in mathematics while convexity is a property especially desired in optimization. Suppose now a piecewise-defined function is convex on each of its defining components - when can we conclude that the entire function is convex? In this paper we provide several convenient, verifiable conditions guaranteeing convexity (or the lack thereof). Sever… ▽ More Functions that are piecewise defined are a common sight in mathematics while convexity is a property especially desired in optimization. Suppose now a piecewise-defined function is convex on each of its defining components - when can we conclude that the entire function is convex? In this paper we provide several convenient, verifiable conditions guaranteeing convexity (or the lack thereof). Several examples are presented to illustrate our results. △ Less

Submitted 16 August, 2014; originally announced August 2014.

MSC Class: 26B25 (Primary) 52A41; 65D17; 90C25 (Secondary)

Optimal rates of convergence of matrices with applications

Authors: Heinz H. Bauschke, J. Y. Bello Cruz, Tran T. A. Nghia, Hung M. Phan, Xianfu Wang

Abstract: We present a systematic study on the linear convergence rates of the powers of (real or complex) matrices. We derive a characterization when the optimal convergence rate is attained. This characterization is given in terms of semi-simpleness of all eigenvalues having the second-largest modulus after 1. We also provide applications of our general results to analyze the optimal convergence rates for… ▽ More We present a systematic study on the linear convergence rates of the powers of (real or complex) matrices. We derive a characterization when the optimal convergence rate is attained. This characterization is given in terms of semi-simpleness of all eigenvalues having the second-largest modulus after 1. We also provide applications of our general results to analyze the optimal convergence rates for several relaxed alternating projection methods and the generalized Douglas-Rachford splitting methods for finding the projection on the intersection of two subspaces. Numerical experiments confirm our convergence analysis. △ Less

Submitted 2 July, 2014; originally announced July 2014.

Comments: 32 pages, 3 figures

MSC Class: 65F10; 65F15

Linear and strong convergence of algorithms involving averaged nonexpansive operators

Authors: Heinz H. Bauschke, Dominikus Noll, Hung M. Phan

Abstract: We introduce regularity notions for averaged nonexpansive operators. Combined with regularity notions of their fixed point sets, we obtain linear and strong convergence results for quasicyclic, cyclic, and random iterations. New convergence results on the Borwein-Tam method (BTM) and on the cylically anchored Douglas-Rachford algorithm (CADRA) are also presented. Finally, we provide a numerical co… ▽ More We introduce regularity notions for averaged nonexpansive operators. Combined with regularity notions of their fixed point sets, we obtain linear and strong convergence results for quasicyclic, cyclic, and random iterations. New convergence results on the Borwein-Tam method (BTM) and on the cylically anchored Douglas-Rachford algorithm (CADRA) are also presented. Finally, we provide a numerical comparison of BTM, CADRA and the classical method of cyclic projections for solving convex feasibility problems. △ Less

Submitted 21 February, 2014; originally announced February 2014.

Comments: 25 pages

MSC Class: Primary 65K05; Secondary 47H09; 90C25

Linear Convergence of the Douglas-Rachford Method for Two Closed Sets

Authors: Hung M. Phan

Abstract: In this paper, we investigate the Douglas-Rachford method for two closed (possibly nonconvex) sets in Euclidean spaces. We show that under certain regularity conditions, the Douglas-Rachford method converges locally with R-linear rate. In convex settings, we prove that the linear convergence is global. Our study recovers recent results on the same topic. In this paper, we investigate the Douglas-Rachford method for two closed (possibly nonconvex) sets in Euclidean spaces. We show that under certain regularity conditions, the Douglas-Rachford method converges locally with R-linear rate. In convex settings, we prove that the linear convergence is global. Our study recovers recent results on the same topic. △ Less

Submitted 19 February, 2015; v1 submitted 25 January, 2014; originally announced January 2014.

MSC Class: Primary 49M27; 65K10; Secondary 47H09; 49J52; 49M20; 49M37; 65K05; 90C26; 90C30

The rate of linear convergence of the Douglas-Rachford algorithm for subspaces is the cosine of the Friedrichs angle

Authors: Heinz H. Bauschke, J. Y. Bello Cruz, Tran T. A. Nghia, Hung M. Phan, Xianfu Wang

Abstract: The Douglas-Rachford splitting algorithm is a classical optimization method that has found many applications. When specialized to two normal cone operators, it yields an algorithm for finding a point in the intersection of two convex sets. This method for solving feasibility problems has attracted a lot of attention due to its good performance even in nonconvex settings. In this paper, we consid… ▽ More The Douglas-Rachford splitting algorithm is a classical optimization method that has found many applications. When specialized to two normal cone operators, it yields an algorithm for finding a point in the intersection of two convex sets. This method for solving feasibility problems has attracted a lot of attention due to its good performance even in nonconvex settings. In this paper, we consider the Douglas-Rachford algorithm for finding a point in the intersection of two subspaces. We prove that the method converges strongly to the projection of the starting point onto the intersection. Moreover, if the sum of the two subspaces is closed, then the convergence is linear with the rate being the cosine of the Friedrichs angle between the subspaces. Our results improve upon existing results in three ways: First, we identify the location of the limit and thus reveal the method as a best approximation algorithm; second, we quantify the rate of convergence, and third, we carry out our analysis in general (possibly infinite-dimensional) Hilbert space. We also provide various examples as well as a comparison with the classical method of alternating projections. △ Less

Submitted 21 December, 2013; v1 submitted 18 September, 2013; originally announced September 2013.

MSC Class: Primary 49M27; 65K05; 65K10; Secondary 47H05; 47H09; 49M37; 90C25

The Method of Alternating Relaxed Projections for two nonconvex sets

Authors: Heinz H. Bauschke, Hung M. Phan, Xianfu Wang

Abstract: The Method of Alternating Projections (MAP), a classical algorithm for solving feasibility prob- lems, has recently been intensely studied for nonconvex sets. However, intrinsically available are only local convergence results: convergence occurs if the starting point is not too far away from solutions to avoid getting trapped in certain regions. Instead of taking full projection steps, it can be… ▽ More The Method of Alternating Projections (MAP), a classical algorithm for solving feasibility prob- lems, has recently been intensely studied for nonconvex sets. However, intrinsically available are only local convergence results: convergence occurs if the starting point is not too far away from solutions to avoid getting trapped in certain regions. Instead of taking full projection steps, it can be advantageous to underrelax, i.e., to move only part way towards the constraint set, in order to enlarge the regions of convergence. In this paper, we thus systematically study the Method of Alternating Relaxed Projections (MARP) for two (possibly nonconvex) sets. Complementing our recent work on MAP, we es- tablish local linear convergence results for the MARP. Several examples illustrate our analysis. △ Less

Submitted 18 May, 2013; originally announced May 2013.

MSC Class: 65K05 (Primary) 47N10; 49J52; 49M20; 65K10; 90C25; 90C26 (Secondary)

Restricted normal cones and sparsity optimization with affine constraints

Authors: Heinz H. Bauschke, D. Russell Luke, Hung M. Phan, Xianfu Wang

Abstract: The problem of finding a vector with the fewest nonzero elements that satisfies an underdetermined system of linear equations is an NP-complete problem that is typically solved numerically via convex heuristics or nicely-behaved non convex relaxations. In this paper we consider the elementary method of alternating projections (MAP) for solving the sparsity optimization problem without employing co… ▽ More The problem of finding a vector with the fewest nonzero elements that satisfies an underdetermined system of linear equations is an NP-complete problem that is typically solved numerically via convex heuristics or nicely-behaved non convex relaxations. In this paper we consider the elementary method of alternating projections (MAP) for solving the sparsity optimization problem without employing convex heuristics. In a parallel paper we recently introduced the restricted normal cone which generalizes the classical Mordukhovich normal cone and reconciles some fundamental gaps in the theory of sufficient conditions for local linear convergence of the MAP algorithm. We use the restricted normal cone together with the notion of superregularity, which is naturally satisfied for the affine sparse optimization problem, to obtain local linear convergence results with estimates for the radius of convergence of the MAP algorithm applied to sparsity optimization with an affine constraint. △ Less

Submitted 2 May, 2012; originally announced May 2012.

MSC Class: 49J52; 49M20; 90C26 (Primary) 15A29; 47H09; 65K05; 65K10; 94A08 (Secondary)

Restricted normal cones and the method of alternating projections

Authors: Heinz H. Bauschke, D. Russell Luke, Hung M. Phan, Xianfu Wang

Abstract: The method of alternating projections (MAP) is a common method for solving feasibility problems. While employed traditionally to subspaces or to convex sets, little was known about the behavior of the MAP in the nonconvex case until 2009, when Lewis, Luke, and Malick derived local linear convergence results provided that a condition involving normal cones holds and at least one of the sets is supe… ▽ More The method of alternating projections (MAP) is a common method for solving feasibility problems. While employed traditionally to subspaces or to convex sets, little was known about the behavior of the MAP in the nonconvex case until 2009, when Lewis, Luke, and Malick derived local linear convergence results provided that a condition involving normal cones holds and at least one of the sets is superregular (a property less restrictive than convexity). However, their results failed to capture very simple classical convex instances such as two lines in three-dimensional space. In this paper, we extend and develop the Lewis-Luke-Malick framework so that not only any two linear subspaces but also any two closed convex sets whose relative interiors meet are covered. We also allow for sets that are more structured such as unions of convex sets. The key tool required is the restricted normal cone, which is a generalization of the classical Mordukhovich normal cone. We thoroughly study restricted normal cones from the viewpoint of constraint qualifications and regularity. Numerous examples are provided to illustrate the theory. △ Less

Submitted 2 May, 2012; originally announced May 2012.

MSC Class: 49J52; 49M20 (Primary) 47H09; 65K05; 65K10; 90C26 (Secondary)

Rated Extremal Principles for Finite and Infinite Systems

Authors: Boris S. Mordukhovich, Hung M. Phan

Abstract: In this paper we introduce new notions of local extremality for finite and infinite systems of closed sets and establish the corresponding extremal principles for them called here rated extremal principles. These developments are in the core geometric theory of variational analysis. We present their applications to calculus and optimality conditions for problems with infinitely many constraints. In this paper we introduce new notions of local extremality for finite and infinite systems of closed sets and establish the corresponding extremal principles for them called here rated extremal principles. These developments are in the core geometric theory of variational analysis. We present their applications to calculus and optimality conditions for problems with infinitely many constraints. △ Less

Submitted 25 February, 2011; originally announced February 2011.

Tangential Extremal Principles for Finite and Infinite Systems of Sets, I: Basic Theory

Authors: Boris S. Mordukhovich, Hung M. Phan

Abstract: In this paper we develop new extremal principles in variational analysis that deal with finite and infinite systems of convex and nonconvex sets. The results obtained, unified under the name of tangential extremal principles, combine primal and dual approaches to the study of variational systems being in fact first extremal principles applied to infinite systems of sets. The first part of the pape… ▽ More In this paper we develop new extremal principles in variational analysis that deal with finite and infinite systems of convex and nonconvex sets. The results obtained, unified under the name of tangential extremal principles, combine primal and dual approaches to the study of variational systems being in fact first extremal principles applied to infinite systems of sets. The first part of the paper concerns the basic theory of tangential extremal principles while the second part presents applications to problems of semi-infinite programming and multiobjective optimization. △ Less

Submitted 21 January, 2011; originally announced January 2011.

Tangential Extremal Principles for Finite and Infinite Systems of Sets, II: Applications to Semi-infinite and Multiobjective Optimization

Authors: Boris S. Mordukhovich, Hung M. Phan

Abstract: This paper contains selected applications of the new tangential extremal principles and related results developed in Part I to calculus rules for infinite intersections of sets and optimality conditions for problems of semi-infinite programming and multiobjective optimization with countable constraints This paper contains selected applications of the new tangential extremal principles and related results developed in Part I to calculus rules for infinite intersections of sets and optimality conditions for problems of semi-infinite programming and multiobjective optimization with countable constraints △ Less

Submitted 21 January, 2011; originally announced January 2011.