Showing 1–5 of 5 results for author: Harrison, M T

Inconsistency of Pitman-Yor process mixtures for the number of components

Authors: Jeffrey W. Miller, Matthew T. Harrison

Abstract: In many applications, a finite mixture is a natural model, but it can be difficult to choose an appropriate number of components. To circumvent this choice, investigators are increasingly turning to Dirichlet process mixtures (DPMs), and Pitman-Yor process mixtures (PYMs), more generally. While these models may be well-suited for Bayesian density estimation, many investigators are using them for i… ▽ More In many applications, a finite mixture is a natural model, but it can be difficult to choose an appropriate number of components. To circumvent this choice, investigators are increasingly turning to Dirichlet process mixtures (DPMs), and Pitman-Yor process mixtures (PYMs), more generally. While these models may be well-suited for Bayesian density estimation, many investigators are using them for inferences about the number of components, by considering the posterior on the number of components represented in the observed data. We show that this posterior is not consistent --- that is, on data from a finite mixture, it does not concentrate at the true number of components. This result applies to a large class of nonparametric mixtures, including DPMs and PYMs, over a wide variety of families of component distributions, including essentially all discrete families, as well as continuous exponential families satisfying mild regularity conditions (such as multivariate Gaussians). △ Less

Submitted 30 August, 2013; originally announced September 2013.

Comments: This is a general treatment of the problem discussed in our related article, "A simple example of Dirichlet process mixture inconsistency for the number of components", Miller and Harrison (2013) arXiv:1301.2708

MSC Class: 62G20 (Primary); 62G05 (Secondary)

Exact sampling and counting for fixed-margin matrices

Authors: Jeffrey W. Miller, Matthew T. Harrison

Abstract: The uniform distribution on matrices with specified row and column sums is often a natural choice of null model when testing for structure in two-way tables (binary or nonnegative integer). Due to the difficulty of sampling from this distribution, many approximate methods have been developed. We will show that by exploiting certain symmetries, exact sampling and counting is in fact possible in man… ▽ More The uniform distribution on matrices with specified row and column sums is often a natural choice of null model when testing for structure in two-way tables (binary or nonnegative integer). Due to the difficulty of sampling from this distribution, many approximate methods have been developed. We will show that by exploiting certain symmetries, exact sampling and counting is in fact possible in many nontrivial real-world cases. We illustrate with real datasets including ecological co-occurrence matrices and contingency tables. △ Less

Submitted 13 August, 2013; v1 submitted 28 January, 2013; originally announced January 2013.

Comments: Published in at http://dx.doi.org/10.1214/13-AOS1131 the Annals of Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical Statistics (http://www.imstat.org). arXiv admin note: text overlap with arXiv:1104.0323

Report number: IMS-AOS-AOS1131

Journal ref: Annals of Statistics 2013, Vol. 41, No. 3, 1569-1592

Importance sampling for weighted binary random matrices with specified margins

Authors: Matthew T. Harrison, Jeffrey W. Miller

Abstract: A sequential importance sampling algorithm is developed for the distribution that results when a matrix of independent, but not identically distributed, Bernoulli random variables is conditioned on a given sequence of row and column sums. This conditional distribution arises in a variety of applications and includes as a special case the uniform distribution over zero-one tables with specified mar… ▽ More A sequential importance sampling algorithm is developed for the distribution that results when a matrix of independent, but not identically distributed, Bernoulli random variables is conditioned on a given sequence of row and column sums. This conditional distribution arises in a variety of applications and includes as a special case the uniform distribution over zero-one tables with specified margins. The algorithm uses dynamic programming to combine hard margin constraints, combinatorial approximations, and additional non-uniform weighting in a principled way to give state-of-the-art results. △ Less

Submitted 16 January, 2013; originally announced January 2013.

Comments: 39 pages (13 pages main text, 26 pages supplementary material); supersedes arXiv:0906.1004

A simple example of Dirichlet process mixture inconsistency for the number of components

Authors: Jeffrey W. Miller, Matthew T. Harrison

Abstract: For data assumed to come from a finite mixture with an unknown number of components, it has become common to use Dirichlet process mixtures (DPMs) not only for density estimation, but also for inferences about the number of components. The typical approach is to use the posterior distribution on the number of components occurring so far --- that is, the posterior on the number of clusters in the o… ▽ More For data assumed to come from a finite mixture with an unknown number of components, it has become common to use Dirichlet process mixtures (DPMs) not only for density estimation, but also for inferences about the number of components. The typical approach is to use the posterior distribution on the number of components occurring so far --- that is, the posterior on the number of clusters in the observed data. However, it turns out that this posterior is not consistent --- it does not converge to the true number of components. In this note, we give an elementary demonstration of this inconsistency in what is perhaps the simplest possible setting: a DPM with normal components of unit variance, applied to data from a "mixture" with one standard normal component. Further, we find that this example exhibits severe inconsistency: instead of going to 1, the posterior probability that there is one cluster goes to 0. △ Less

Submitted 12 January, 2013; originally announced January 2013.

MSC Class: 62G20; 62G05

Estimation of the Rate-Distortion Function

Authors: M. T. Harrison, I. Kontoyiannis

Abstract: Motivated by questions in lossy data compression and by theoretical considerations, we examine the problem of estimating the rate-distortion function of an unknown (not necessarily discrete-valued) source from empirical data. Our focus is the behavior of the so-called "plug-in" estimator, which is simply the rate-distortion function of the empirical distribution of the observed data. Sufficient… ▽ More Motivated by questions in lossy data compression and by theoretical considerations, we examine the problem of estimating the rate-distortion function of an unknown (not necessarily discrete-valued) source from empirical data. Our focus is the behavior of the so-called "plug-in" estimator, which is simply the rate-distortion function of the empirical distribution of the observed data. Sufficient conditions are given for its consistency, and examples are provided to demonstrate that in certain cases it fails to converge to the true rate-distortion function. The analysis of its performance is complicated by the fact that the rate-distortion function is not continuous in the source distribution; the underlying mathematical problem is closely related to the classical problem of establishing the consistency of maximum likelihood estimators. General consistency results are given for the plug-in estimator applied to a broad class of sources, including all stationary and ergodic ones. A more general class of estimation problems is also considered, arising in the context of lossy data compression when the allowed class of coding distributions is restricted; analogous results are developed for the plug-in estimator in that case. Finally, consistency theorems are formulated for modified (e.g., penalized) versions of the plug-in, and for estimating the optimal reproduction distribution. △ Less

Submitted 11 April, 2008; v1 submitted 2 February, 2007; originally announced February 2007.

Comments: 18 pages, no figures [v2: removed an example with an error; corrected typos; a shortened version will appear in IEEE Trans. Inform. Theory]

Journal ref: IEEE Transactions on Information Theory, 54 (2008): 3757-3762