Showing 1–6 of 6 results for author: Rush, K

Improved Differential Privacy for SGD via Optimal Private Linear Operators on Adaptive Streams

Authors: Sergey Denisov, Brendan McMahan, Keith Rush, Adam Smith, Abhradeep Guha Thakurta

Abstract: Motivated by recent applications requiring differential privacy over adaptive streams, we investigate the question of optimal instantiations of the matrix mechanism in this setting. We prove fundamental theoretical results on the applicability of matrix factorizations to adaptive streams, and provide a parameter-free fixed-point algorithm for computing optimal factorizations. We instantiate this f… ▽ More Motivated by recent applications requiring differential privacy over adaptive streams, we investigate the question of optimal instantiations of the matrix mechanism in this setting. We prove fundamental theoretical results on the applicability of matrix factorizations to adaptive streams, and provide a parameter-free fixed-point algorithm for computing optimal factorizations. We instantiate this framework with respect to concrete matrices which arise naturally in machine learning, and train user-level differentially private models with the resulting optimal mechanisms, yielding significant improvements in a notable problem in federated learning with user-level differential privacy. △ Less

Submitted 17 January, 2023; v1 submitted 16 February, 2022; originally announced February 2022.

Comments: 33 pages, 6 figures. Associated code at https://github.com/google-research/federated/tree/master/dp_matrix_factorization

Iterated Vector Fields and Conservatism, with Applications to Federated Learning

Authors: Zachary Charles, Keith Rush

Abstract: We study whether iterated vector fields (vector fields composed with themselves) are conservative. We give explicit examples of vector fields for which this self-composition preserves conservatism. Notably, this includes gradient vector fields of loss functions associated with some generalized linear models. As we show, characterizing the set of vector fields satisfying this condition leads to non… ▽ More We study whether iterated vector fields (vector fields composed with themselves) are conservative. We give explicit examples of vector fields for which this self-composition preserves conservatism. Notably, this includes gradient vector fields of loss functions associated with some generalized linear models. As we show, characterizing the set of vector fields satisfying this condition leads to non-trivial geometric questions. In the context of federated learning, we show that when clients have loss functions whose gradients satisfy this condition, federated averaging is equivalent to gradient descent on a surrogate loss function. We leverage this to derive novel convergence results for federated learning. By contrast, we demonstrate that when the client losses violate this property, federated averaging can yield behavior which is fundamentally distinct from centralized optimization. Finally, we discuss theoretical and practical questions our analytical framework raises for federated learning. △ Less

Submitted 12 November, 2021; v1 submitted 8 September, 2021; originally announced September 2021.

Adaptive Federated Optimization

Authors: Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, H. Brendan McMahan

Abstract: Federated learning is a distributed machine learning paradigm in which a large number of clients coordinate with a central server to learn a model without sharing their own training data. Standard federated optimization methods such as Federated Averaging (FedAvg) are often difficult to tune and exhibit unfavorable convergence behavior. In non-federated settings, adaptive optimization methods have… ▽ More Federated learning is a distributed machine learning paradigm in which a large number of clients coordinate with a central server to learn a model without sharing their own training data. Standard federated optimization methods such as Federated Averaging (FedAvg) are often difficult to tune and exhibit unfavorable convergence behavior. In non-federated settings, adaptive optimization methods have had notable success in combating such issues. In this work, we propose federated versions of adaptive optimizers, including Adagrad, Adam, and Yogi, and analyze their convergence in the presence of heterogeneous data for general non-convex settings. Our results highlight the interplay between client heterogeneity and communication efficiency. We also perform extensive experiments on these methods and show that the use of adaptive optimizers can significantly improve the performance of federated learning. △ Less

Submitted 8 September, 2021; v1 submitted 29 February, 2020; originally announced March 2020.

Comments: Published as a conference paper at ICLR 2021

Randomized Verblunsky Parameters in Steklov's Problem

Authors: Keith Rush

Abstract: We consider randomized Verblunsky parameters for orthogonal polynomials on the unit circle as they relate to the problem of Steklov, bounding the polynomials' uniform norm independent of $n$. We consider randomized Verblunsky parameters for orthogonal polynomials on the unit circle as they relate to the problem of Steklov, bounding the polynomials' uniform norm independent of $n$. △ Less

Submitted 28 May, 2017; v1 submitted 5 April, 2017; originally announced April 2017.

Comments: 12 pages

MSC Class: 42C05

Journal ref: Journal of Mathematical Analysis and Applications 468 (2), 608-621. 2018

On Schur parameters in Steklov's problem

Authors: S. Denisov, K. Rush

Abstract: We study the recursion (aka Schur) parameters for monic polynomials orthogonal on the unit circle with respect to a weight which provides negative answer to the conjecture of Steklov. We study the recursion (aka Schur) parameters for monic polynomials orthogonal on the unit circle with respect to a weight which provides negative answer to the conjecture of Steklov. △ Less

Submitted 14 June, 2016; originally announced June 2016.

MSC Class: 33C45

Orthogonal polynomials on the circle for the weight w satisfying conditions: w,1/w in BMO(T)

Authors: S. Denisov, K. Rush

Abstract: In the case when the weight and its inverse belong to BMO(T), we prove the asymptotics of the monic orthogonal polynomials in L^p, 2<p<p_0. Immediate applications include the estimates on the uniform norm and asymptotics for the polynomial entropy. In the case when the weight and its inverse belong to BMO(T), we prove the asymptotics of the monic orthogonal polynomials in L^p, 2<p<p_0. Immediate applications include the estimates on the uniform norm and asymptotics for the polynomial entropy. △ Less

Submitted 1 November, 2016; v1 submitted 13 January, 2016; originally announced January 2016.

Comments: Lemma 3.1 and its proof are corrected. Minor changes are made as consequences of these corrections in proofs of Lemma 3.2 and 3.4