Title:Estimating Frequency Statistics through Distinct Count Measurements

Abstract:One of the most common statistics computed over data elements is the number of distinct keys. It is widely used in practice for a diverse set of applications. A thread of research pioneered by Flajolet and Martin three decades ago initiated the design of highly effective approximate counting structures, which with size that is double logarithmic in the number of keys provides estimates with a small relative error. Moreover, the structures are mergeable/composable, which makes them suitable for streamed, parallel, or distributed computation.
We present a novel framework that leverages these structures and their existing implementations to approximate any frequency statistics of the key frequency distribution, where a contribution of a key to the aggregate is concave and grows (sub)linearly with its frequency in the data. These fundamental aggregations are very common in text, graphs, and logs analysis and include logarithms, low frequency moments, and capping statistics common in online advertising.
Our scheme is powerful but very simple to apply: We specify tailored mapping functions of data elements to output elements. The functions can be applied independently to each element and are chosen so that we can recover our statistics on data elements from distinct statistics of output elements. In a nut shell, we specify the {\em complement Laplace} transform of the input frequencies and show how it can be approximated pointwise using such distinct count measurements. We then derive corresponding inverse transforms for the target statistics. Our work provides optimal asymptotic bounds together with highly practical algorithms for these widely used statistics.