Skip to main content

Showing 1–4 of 4 results for author: Ramaswamy, S

Searching in archive stat. Search in all archives.
.
  1. arXiv:2009.10031  [pdf, other

    cs.LG cs.CR stat.ML

    Training Production Language Models without Memorizing User Data

    Authors: Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H. Brendan McMahan, Françoise Beaufays

    Abstract: This paper presents the first consumer-scale next-word prediction (NWP) model trained with Federated Learning (FL) while leveraging the Differentially Private Federated Averaging (DP-FedAvg) technique. There has been prior work on building practical FL infrastructure, including work demonstrating the feasibility of training language models on mobile devices using such infrastructure. It has also b… ▽ More

    Submitted 21 September, 2020; originally announced September 2020.

  2. arXiv:2006.07490  [pdf, other

    cs.LG cs.CL stat.ML

    Understanding Unintended Memorization in Federated Learning

    Authors: Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, Françoise Beaufays

    Abstract: Recent works have shown that generative sequence models (e.g., language models) have a tendency to memorize rare or unique sequences in the training data. Since useful models are often trained on sensitive data, to ensure the privacy of the training data it is critical to identify and mitigate such unintended memorization. Federated Learning (FL) has emerged as a novel framework for large-scale di… ▽ More

    Submitted 12 June, 2020; originally announced June 2020.

  3. arXiv:1911.06679  [pdf, other

    cs.LG stat.ML

    Generative Models for Effective ML on Private, Decentralized Datasets

    Authors: Sean Augenstein, H. Brendan McMahan, Daniel Ramage, Swaroop Ramaswamy, Peter Kairouz, Mingqing Chen, Rajiv Mathews, Blaise Aguera y Arcas

    Abstract: To improve real-world applications of machine learning, experienced modelers develop intuition about their datasets, their models, and how the two interact. Manual inspection of raw data - of representative samples, of outliers, of misclassifications - is an essential tool in a) identifying and fixing problems in the data, b) generating new modeling hypotheses, and c) assigning or refining human-p… ▽ More

    Submitted 4 February, 2020; v1 submitted 15 November, 2019; originally announced November 2019.

    Comments: 26 pages, 8 figures. Camera-ready ICLR 2020 version

  4. arXiv:1905.03871  [pdf, other

    cs.LG stat.ML

    Differentially Private Learning with Adaptive Clipping

    Authors: Galen Andrew, Om Thakkar, H. Brendan McMahan, Swaroop Ramaswamy

    Abstract: Existing approaches for training neural networks with user-level differential privacy (e.g., DP Federated Averaging) in federated learning (FL) settings involve bounding the contribution of each user's model update by clipping it to some constant value. However there is no good a priori setting of the clipping norm across tasks and learning settings: the update norm distribution depends on the mod… ▽ More

    Submitted 9 May, 2022; v1 submitted 9 May, 2019; originally announced May 2019.

    Comments: Accepted to NeurIPS, 2021