-
The Saddle-Point Accountant for Differential Privacy
Authors:
Wael Alghamdi,
Shahab Asoodeh,
Flavio P. Calmon,
Juan Felipe Gomez,
Oliver Kosut,
Lalitha Sankar,
Fei Wei
Abstract:
We introduce a new differential privacy (DP) accountant called the saddle-point accountant (SPA). SPA approximates privacy guarantees for the composition of DP mechanisms in an accurate and fast manner. Our approach is inspired by the saddle-point method -- a ubiquitous numerical technique in statistics. We prove rigorous performance guarantees by deriving upper and lower bounds for the approximat…
▽ More
We introduce a new differential privacy (DP) accountant called the saddle-point accountant (SPA). SPA approximates privacy guarantees for the composition of DP mechanisms in an accurate and fast manner. Our approach is inspired by the saddle-point method -- a ubiquitous numerical technique in statistics. We prove rigorous performance guarantees by deriving upper and lower bounds for the approximation error offered by SPA. The crux of SPA is a combination of large-deviation methods with central limit theorems, which we derive via exponentially tilting the privacy loss random variables corresponding to the DP mechanisms. One key advantage of SPA is that it runs in constant time for the $n$-fold composition of a privacy mechanism. Numerical experiments demonstrate that SPA achieves comparable accuracy to state-of-the-art accounting methods with a faster runtime.
△ Less
Submitted 19 August, 2022;
originally announced August 2022.
-
Measuring Information from Moments
Authors:
Wael Alghamdi,
Flavio P. Calmon
Abstract:
We investigate the problem of representing information measures in terms of the moments of the underlying random variables. First, we derive polynomial approximations of the conditional expectation operator. We then apply these approximations to bound the best mean-square error achieved by a polynomial estimator -- referred to here as the PMMSE. In Gaussian channels, the PMMSE coincides with the m…
▽ More
We investigate the problem of representing information measures in terms of the moments of the underlying random variables. First, we derive polynomial approximations of the conditional expectation operator. We then apply these approximations to bound the best mean-square error achieved by a polynomial estimator -- referred to here as the PMMSE. In Gaussian channels, the PMMSE coincides with the minimum mean-square error (MMSE) if and only if the input is either Gaussian or constant, i.e., if and only if the conditional expectation of the input of the channel given the output is a polynomial of degree at most 1. By combining the PMMSE with the I-MMSE relationship, we derive new formulas for information measures (e.g., differential entropy, mutual information) that are given in terms of the moments of the underlying random variables. As an application, we introduce estimators for information measures from data via approximating the moments in our formulas by sample moments. These estimators are shown to be asymptotically consistent and possess desirable properties, e.g., invariance to affine transformations when used to estimate mutual information.
△ Less
Submitted 1 September, 2021;
originally announced September 2021.
-
Polynomial Approximations of Conditional Expectations in Scalar Gaussian Channels
Authors:
Wael Alghamdi,
Flavio P. Calmon
Abstract:
We consider a channel $Y=X+N$ where $X$ is a random variable satisfying $\mathbb{E}[|X|]<\infty$ and $N$ is an independent standard normal random variable. We show that the minimum mean-square error estimator of $X$ from $Y,$ which is given by the conditional expectation $\mathbb{E}[X \mid Y],$ is a polynomial in $Y$ if and only if it is linear or constant; these two cases correspond to $X$ being…
▽ More
We consider a channel $Y=X+N$ where $X$ is a random variable satisfying $\mathbb{E}[|X|]<\infty$ and $N$ is an independent standard normal random variable. We show that the minimum mean-square error estimator of $X$ from $Y,$ which is given by the conditional expectation $\mathbb{E}[X \mid Y],$ is a polynomial in $Y$ if and only if it is linear or constant; these two cases correspond to $X$ being Gaussian or a constant, respectively. We also prove that the higher-order derivatives of $y \mapsto \mathbb{E}[X \mid Y=y]$ are expressible as multivariate polynomials in the functions $y \mapsto \mathbb{E}\left[ \left( X - \mathbb{E}[X \mid Y] \right)^k \mid Y = y \right]$ for $k\in \mathbb{N}.$ These expressions yield bounds on the $2$-norm of the derivatives of the conditional expectation. These bounds imply that, if $X$ has a compactly-supported density that is even and decreasing on the positive half-line, then the error in approximating the conditional expectation $\mathbb{E}[X \mid Y]$ by polynomials in $Y$ of degree at most $n$ decays faster than any polynomial in $n.$
△ Less
Submitted 11 February, 2021;
originally announced February 2021.
