Title:Phase Transitions for High Dimensional Clustering and Related Problems

Abstract:Consider a two-class clustering problem where we have $X_i = \ell_i \mu + Z_i$, $Z_i \stackrel{iid}{\sim} N(0, I_p)$, $1 \leq i \leq n$. The class labels $\ell_i$ are unknown and the main interest is to estimate them. The feature vector $\mu$ is also unknown but is presumably sparse in that only a small fraction of the features are useful for clustering.
We are interested in the fundamental limits. In the two-dimensional phase space calibrating the rarity of the useful features and their strengths, we identify the separating boundary for Region of Impossibility and Region of Possibility. In the former, the useful features are too rare/weak to allow successful clustering. In the latter, the useful features are strong enough and successful clustering is possible.
We propose both classical PCA and Important Features PCA (IF-PCA) for clustering. We also propose two aggregation methods for clustering. For any parameter in the Region of Possibility, one or more of these four methods yield successful clustering.
We extend the fundamental limits of clustering successfully to many cases where the noise is colored, using Le Cam's idea on comparison of experiments. We also extend the study to two closely related problems: the signal recovery problem where the interest is to recover the support of $\mu$, and the hypothesis testing problem where we test whether $X_i$ are samples from $N(0, I_p)$ or are generated according to the model above. We compare the fundamental limits for all three problems and expose interesting insight. We also find an interesting phase transition for IF-PCA.
Our results require delicate analysis, especially on post-selection Random Matrix Theory and on lower bound arguments.