-
Scalable Distributed Optimization of Multi-Dimensional Functions Despite Byzantine Adversaries
Authors:
Kananart Kuwaranancharoen,
Lei Xin,
Shreyas Sundaram
Abstract:
The problem of distributed optimization requires a group of networked agents to compute a parameter that minimizes the average of their local cost functions. While there are a variety of distributed optimization algorithms that can solve this problem, they are typically vulnerable to "Byzantine" agents that do not follow the algorithm. Recent attempts to address this issue focus on single dimensio…
▽ More
The problem of distributed optimization requires a group of networked agents to compute a parameter that minimizes the average of their local cost functions. While there are a variety of distributed optimization algorithms that can solve this problem, they are typically vulnerable to "Byzantine" agents that do not follow the algorithm. Recent attempts to address this issue focus on single dimensional functions, or assume certain statistical properties of the functions at the agents. In this paper, we provide two resilient, scalable, distributed optimization algorithms for multi-dimensional functions. Our schemes involve two filters, (1) a distance-based filter and (2) a min-max filter, which each remove neighborhood states that are extreme (defined precisely in our algorithms) at each iteration. We show that these algorithms can mitigate the impact of up to $F$ (unknown) Byzantine agents in the neighborhood of each regular agent. In particular, we show that if the network topology satisfies certain conditions, all of the regular agents' states are guaranteed to converge to a bounded region that contains the minimizer of the average of the regular agents' functions.
△ Less
Submitted 14 March, 2024; v1 submitted 11 March, 2024;
originally announced March 2024.
-
The Minimizer of the Sum of Two Strongly Convex Functions
Authors:
Kananart Kuwaranancharoen,
Shreyas Sundaram
Abstract:
The optimization problem concerning the determination of the minimizer for the sum of convex functions holds significant importance in the realm of distributed and decentralized optimization. In scenarios where full knowledge of the functions is not available, limiting information to individual minimizers and convexity parameters -- either due to privacy concerns or the nature of solution analysis…
▽ More
The optimization problem concerning the determination of the minimizer for the sum of convex functions holds significant importance in the realm of distributed and decentralized optimization. In scenarios where full knowledge of the functions is not available, limiting information to individual minimizers and convexity parameters -- either due to privacy concerns or the nature of solution analysis -- necessitates an exploration of the region encompassing potential minimizers based solely on these known quantities. The characterization of this region becomes notably intricate when dealing with multivariate strongly convex functions compared to the univariate case. This paper contributes outer and inner approximations for the region harboring the minimizer of the sum of two strongly convex functions, given a constraint on the norm of the gradient at the minimizer of the sum. Notably, we explicitly delineate the boundaries and interiors of both the outer and inner approximations. Intriguingly, the boundaries as well as the interiors turn out to be identical. Furthermore, we establish that the boundary of the region containing potential minimizers aligns with that of the outer and inner approximations.
△ Less
Submitted 21 September, 2024; v1 submitted 22 May, 2023;
originally announced May 2023.
-
On the Geometric Convergence of Byzantine-Resilient Distributed Optimization Algorithms
Authors:
Kananart Kuwaranancharoen,
Shreyas Sundaram
Abstract:
The problem of designing distributed optimization algorithms that are resilient to Byzantine adversaries has received significant attention. For the Byzantine-resilient distributed optimization problem, the goal is to (approximately) minimize the average of the local cost functions held by the regular (non adversarial) agents in the network. In this paper, we provide a general algorithmic framewor…
▽ More
The problem of designing distributed optimization algorithms that are resilient to Byzantine adversaries has received significant attention. For the Byzantine-resilient distributed optimization problem, the goal is to (approximately) minimize the average of the local cost functions held by the regular (non adversarial) agents in the network. In this paper, we provide a general algorithmic framework for Byzantine-resilient distributed optimization which includes some state-of-the-art algorithms as special cases. We analyze the convergence of algorithms within the framework, and derive a geometric rate of convergence of all regular agents to a ball around the optimal solution (whose size we characterize). Furthermore, we show that approximate consensus can be achieved geometrically fast under some minimal conditions. Our analysis provides insights into the relationship among the convergence region, distance between regular agents' values, step-size, and properties of the agents' functions for Byzantine-resilient distributed optimization.
△ Less
Submitted 25 December, 2024; v1 submitted 18 May, 2023;
originally announced May 2023.
-
On the Conditional Expectation of Mean Shifted Gaussian Distributions
Authors:
Kananart Kuwaranancharoen
Abstract:
In this paper, we consider a property of univariate Gaussian distributions namely conditional expectation shift (or centroid shift). Specifically, we compare two Gaussian distributions in which they differ only in their means. Equivalently, we can view this situation as one of the distribution is shifted to the right. These two distributions are conditioned on the same event in which the realizati…
▽ More
In this paper, we consider a property of univariate Gaussian distributions namely conditional expectation shift (or centroid shift). Specifically, we compare two Gaussian distributions in which they differ only in their means. Equivalently, we can view this situation as one of the distribution is shifted to the right. These two distributions are conditioned on the same event in which the realizations fall in the right interval or left interval. We show that if a Gaussian distribution is shifted to the right while the conditioning event remains the same then the conditional expectation is shifted to the right concurrently.
△ Less
Submitted 9 June, 2020;
originally announced June 2020.
-
Byzantine-Resilient Distributed Optimization of Multi-Dimensional Functions
Authors:
Kananart Kuwaranancharoen,
Lei Xin,
Shreyas Sundaram
Abstract:
The problem of distributed optimization requires a group of agents to reach agreement on a parameter that minimizes the average of their local cost functions using information received from their neighbors. While there are a variety of distributed optimization algorithms that can solve this problem, they are typically vulnerable to malicious (or ``Byzantine'') agents that do not follow the algorit…
▽ More
The problem of distributed optimization requires a group of agents to reach agreement on a parameter that minimizes the average of their local cost functions using information received from their neighbors. While there are a variety of distributed optimization algorithms that can solve this problem, they are typically vulnerable to malicious (or ``Byzantine'') agents that do not follow the algorithm. Recent attempts to address this issue focus on single dimensional functions, or provide analysis under certain assumptions on the statistical properties of the functions at the agents. In this paper, we propose a resilient distributed optimization algorithm for multi-dimensional convex functions. Our scheme involves two filtering steps at each iteration of the algorithm: (1) distance-based and (2) component-wise removal of extreme states. We show that this algorithm can mitigate the impact of up to $F$ Byzantine agents in the neighborhood of each regular node, without knowing the identities of the Byzantine agents in advance. In particular, we show that if the network topology satisfies certain conditions, all of the regular states are guaranteed to asymptotically converge to a bounded region that contains the global minimizer.
△ Less
Submitted 4 March, 2024; v1 submitted 19 March, 2020;
originally announced March 2020.
-
On the Set of Possible Minimizers of a Sum of Known and Unknown Functions
Authors:
Kananart Kuwaranancharoen,
Shreyas Sundaram
Abstract:
The problem of finding the minimizer of a sum of convex functions is central to the field of optimization. Thus, it is of interest to understand how that minimizer is related to the properties of the individual functions in the sum. In this paper, we consider the scenario where one of the individual functions in the sum is not known completely. Instead, only a region containing the minimizer of th…
▽ More
The problem of finding the minimizer of a sum of convex functions is central to the field of optimization. Thus, it is of interest to understand how that minimizer is related to the properties of the individual functions in the sum. In this paper, we consider the scenario where one of the individual functions in the sum is not known completely. Instead, only a region containing the minimizer of the unknown function is known, along with some general characteristics (such as strong convexity parameters). Given this limited information about a portion of the overall function, we provide a necessary condition which can be used to construct an upper bound on the region containing the minimizer of the sum of known and unknown functions. We provide this necessary condition in both the general case where the uncertainty region of the minimizer of the unknown function is arbitrary, and in the specific case where the uncertainty region is a ball.
△ Less
Submitted 19 March, 2020;
originally announced March 2020.
-
On the Location of the Minimizer of the Sum of Two Strongly Convex Functions
Authors:
Kananart Kuwaranancharoen,
Shreyas Sundaram
Abstract:
The problem of finding the minimizer of a sum of convex functions is central to the field of distributed optimization. Thus, it is of interest to understand how that minimizer is related to the properties of the individual functions in the sum. In this paper, we provide an upper bound on the region containing the minimizer of the sum of two strongly convex functions. We consider two scenarios with…
▽ More
The problem of finding the minimizer of a sum of convex functions is central to the field of distributed optimization. Thus, it is of interest to understand how that minimizer is related to the properties of the individual functions in the sum. In this paper, we provide an upper bound on the region containing the minimizer of the sum of two strongly convex functions. We consider two scenarios with different constraints on the upper bound of the gradients of the functions. In the first scenario, the gradient constraint is imposed on the location of the potential minimizer, while in the second scenario, the gradient constraint is imposed on a given convex set in which the minimizers of two original functions are embedded. We characterize the boundaries of the regions containing the minimizer in both scenarios.
△ Less
Submitted 3 December, 2018; v1 submitted 20 April, 2018;
originally announced April 2018.
