Showing 1–5 of 5 results for author: Sarbu, S

On learning parametric distributions from quantized samples

Authors: Septimia Sarbu, Abdellatif Zaidi

Abstract: We consider the problem of learning parametric distributions from their quantized samples in a network. Specifically, $n$ agents or sensors observe independent samples of an unknown parametric distribution; and each of them uses $k$ bits to describe its observed sample to a central processor whose goal is to estimate the unknown distribution. First, we establish a generalization of the well-known… ▽ More We consider the problem of learning parametric distributions from their quantized samples in a network. Specifically, $n$ agents or sensors observe independent samples of an unknown parametric distribution; and each of them uses $k$ bits to describe its observed sample to a central processor whose goal is to estimate the unknown distribution. First, we establish a generalization of the well-known van Trees inequality to general $L_p$-norms, with $p > 1$, in terms of Generalized Fisher information. Then, we develop minimax lower bounds on the estimation error for two losses: general $L_p$-norms and the related Wasserstein loss from optimal transport. △ Less

Submitted 21 July, 2022; v1 submitted 25 May, 2021; originally announced May 2021.

Comments: Short version accepted for publication at the IEEE Information Theory Symposium (ISIT) 2021; this version contains the detailed proofs with some minor corrections

Learning in Variational Autoencoders with Kullback-Leibler and Renyi Integral Bounds

Authors: Septimia Sârbu, Riccardo Volpi, Alexandra Peşte, Luigi Malagò

Abstract: In this paper we propose two novel bounds for the log-likelihood based on Kullback-Leibler and the Rényi divergences, which can be used for variational inference and in particular for the training of Variational AutoEncoders. Our proposal is motivated by the difficulties encountered in training VAEs on continuous datasets with high contrast images, such as those with handwritten digits and charact… ▽ More In this paper we propose two novel bounds for the log-likelihood based on Kullback-Leibler and the Rényi divergences, which can be used for variational inference and in particular for the training of Variational AutoEncoders. Our proposal is motivated by the difficulties encountered in training VAEs on continuous datasets with high contrast images, such as those with handwritten digits and characters, where numerical issues often appear unless noise is added, either to the dataset during training or to the generative model given by the decoder. The new bounds we propose, which are obtained from the maximization of the likelihood of an interval for the observations, allow numerically stable training procedures without the necessity of adding any extra source of noise to the data. △ Less

Submitted 5 July, 2018; originally announced July 2018.

Comments: accepted at the ICML 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models, Stockholm, Sweden, 2018

On the Courtade-Kumar conjecture for certain classes of Boolean functions

Authors: Septimia Sarbu

Abstract: We prove the Courtade-Kumar conjecture, for certain classes of $n$-dimensional Boolean functions, $\forall n\geq 2$ and for all values of the error probability of the binary symmetric channel, $\forall 0 \leq p \leq \frac{1}{2}$. Let $\mathbf{X}=[X_1...X_n]$ be a vector of independent and identically distributed Bernoulli$(\frac{1}{2})$ random variables, which are the input to a memoryless binary… ▽ More We prove the Courtade-Kumar conjecture, for certain classes of $n$-dimensional Boolean functions, $\forall n\geq 2$ and for all values of the error probability of the binary symmetric channel, $\forall 0 \leq p \leq \frac{1}{2}$. Let $\mathbf{X}=[X_1...X_n]$ be a vector of independent and identically distributed Bernoulli$(\frac{1}{2})$ random variables, which are the input to a memoryless binary symmetric channel, with the error probability in the interval $0 \leq p \leq \frac{1}{2}$, and $\mathbf{Y}=[Y_1...Y_n]$ the corresponding output. Let $f:\{0,1\}^n \rightarrow \{0,1\}$ be an $n$-dimensional Boolean function. Then, the Courtade-Kumar conjecture states that the mutual information $\operatorname{MI}(f(\mathbf{X}),\mathbf{Y}) \leq 1-\operatorname{H}(p)$, where $\operatorname{H}(p)$ is the binary entropy function. △ Less

Submitted 13 February, 2017; originally announced February 2017.

Comments: submitted to 2017 IEEE International Symposium on Information Theory (ISIT 2017); this article is a summarized version of my previous arXiv manuscript arXiv:1701.05014 , which was rejected by the IEEE Transactions on Information Theory

The maximum mutual information between the output of a discrete symmetric channel and several classes of Boolean functions of its input

Authors: Septimia Sarbu

Abstract: We prove the Courtade-Kumar conjecture, for several classes of n-dimensional Boolean functions, for all $n \geq 2$ and for all values of the error probability of the binary symmetric channel, $0 \leq p \leq 1/2$. This conjecture states that the mutual information between any Boolean function of an n-dimensional vector of independent and identically distributed inputs to a memoryless binary symmetr… ▽ More We prove the Courtade-Kumar conjecture, for several classes of n-dimensional Boolean functions, for all $n \geq 2$ and for all values of the error probability of the binary symmetric channel, $0 \leq p \leq 1/2$. This conjecture states that the mutual information between any Boolean function of an n-dimensional vector of independent and identically distributed inputs to a memoryless binary symmetric channel and the corresponding vector of outputs is upper-bounded by $1-\operatorname{H}(p)$, where $\operatorname{H}(p)$ represents the binary entropy function. That is, let $\mathbf{X}=[X_1 \ldots X_n]$ be a vector of independent and identically distributed Bernoulli(1/2) random variables, which are the input to a memoryless binary symmetric channel, with the error probability in the interval $0 \leq p \leq 1/2$ and $\mathbf{Y}=[Y_1 \ldots Y_n]$ the corresponding output. Let $f:\{0,1\}^n \rightarrow \{0,1\}$ be an n-dimensional Boolean function. Then, $\operatorname{MI}(f(X),Y) \leq 1-\operatorname{H}(p)$. Our proof employs Karamata's theorem, concepts from probability theory, transformations of random variables and vectors and algebraic manipulations. △ Less

Submitted 8 February, 2017; v1 submitted 18 January, 2017; originally announced January 2017.

Comments: Withdrawn due to rejection by the IEEE Transactions on Information Theory; the reason for rejection was limited originality and scope of the proposed result

The maximum mutual information between the output of a binary symmetric channel and a Boolean function of its input

Authors: Septimia Sarbu

Abstract: We prove the Courtade-Kumar conjecture, which states that the mutual information between any Boolean function of an $n$-dimensional vector of independent and identically distributed inputs to a memoryless binary symmetric channel and the corresponding vector of outputs is upper-bounded by $1-\operatorname{H}(p)$, where $\operatorname{H}(p)$ represents the binary entropy function. That is, let… ▽ More We prove the Courtade-Kumar conjecture, which states that the mutual information between any Boolean function of an $n$-dimensional vector of independent and identically distributed inputs to a memoryless binary symmetric channel and the corresponding vector of outputs is upper-bounded by $1-\operatorname{H}(p)$, where $\operatorname{H}(p)$ represents the binary entropy function. That is, let $\mathbf{X}=[X_1...X_n]$ be a vector of independent and identically distributed Bernoulli($1/2$) random variables, which are the input to a memoryless binary symmetric channel, with the error probability equal to $0 \leq p \leq 1/2$, and $\mathbf{Y}=[Y_1...Y_n]$ the corresponding output. Let $f:\{0,1\}^n \rightarrow \{0,1\}$ be an $n$-dimensional Boolean function. Then, $\operatorname{MI}(f(\mathbf{X}),\mathbf{Y}) \leq 1-\operatorname{H}(p)$. We provide the proof for the most general case of the conjecture, that is for any $n$-dimensional Boolean function $f$ and for any value of the error probability of the binary symmetric channel, $0 \leq p \leq 1/2$. Our proof employs only basic concepts from information theory, probability theory and transformations of random variables and vectors. △ Less

Submitted 16 January, 2017; v1 submitted 18 April, 2016; originally announced April 2016.

Comments: This paper has been withdrawn due to a critical error in the way the equation of the mutual information that involved conditional entropies was applied