-
Ground Truth Bias in External Cluster Validity Indices
Authors:
Yang Lei,
James C. Bezdek,
Simone Romano,
Nguyen Xuan Vinh,
Jeffrey Chan,
James Bailey
Abstract:
It has been noticed that some external CVIs exhibit a preferential bias towards a larger or smaller number of clusters which is monotonic (directly or inversely) in the number of clusters in candidate partitions. This type of bias is caused by the functional form of the CVI model. For example, the popular Rand index (RI) exhibits a monotone increasing (NCinc) bias, while the Jaccard Index (JI) ind…
▽ More
It has been noticed that some external CVIs exhibit a preferential bias towards a larger or smaller number of clusters which is monotonic (directly or inversely) in the number of clusters in candidate partitions. This type of bias is caused by the functional form of the CVI model. For example, the popular Rand index (RI) exhibits a monotone increasing (NCinc) bias, while the Jaccard Index (JI) index suffers from a monotone decreasing (NCdec) bias. This type of bias has been previously recognized in the literature. In this work, we identify a new type of bias arising from the distribution of the ground truth (reference) partition against which candidate partitions are compared. We call this new type of bias ground truth (GT) bias. This type of bias occurs if a change in the reference partition causes a change in the bias status (e.g., NCinc, NCdec) of a CVI. For example, NCinc bias in the RI can be changed to NCdec bias by skewing the distribution of clusters in the ground truth partition. It is important for users to be aware of this new type of biased behaviour, since it may affect the interpretations of CVI results. The objective of this article is to study the empirical and theoretical implications of GT bias. To the best of our knowledge, this is the first extensive study of such a property for external cluster validity indices.
△ Less
Submitted 17 June, 2016;
originally announced June 2016.
-
Adjusting for Chance Clustering Comparison Measures
Authors:
Simone Romano,
Nguyen Xuan Vinh,
James Bailey,
Karin Verspoor
Abstract:
Adjusted for chance measures are widely used to compare partitions/clusterings of the same data set. In particular, the Adjusted Rand Index (ARI) based on pair-counting, and the Adjusted Mutual Information (AMI) based on Shannon information theory are very popular in the clustering community. Nonetheless it is an open problem as to what are the best application scenarios for each measure and guide…
▽ More
Adjusted for chance measures are widely used to compare partitions/clusterings of the same data set. In particular, the Adjusted Rand Index (ARI) based on pair-counting, and the Adjusted Mutual Information (AMI) based on Shannon information theory are very popular in the clustering community. Nonetheless it is an open problem as to what are the best application scenarios for each measure and guidelines in the literature for their usage are sparse, with the result that users often resort to using both. Generalized Information Theoretic (IT) measures based on the Tsallis entropy have been shown to link pair-counting and Shannon IT measures. In this paper, we aim to bridge the gap between adjustment of measures based on pair-counting and measures based on information theory. We solve the key technical challenge of analytically computing the expected value and variance of generalized IT measures. This allows us to propose adjustments of generalized IT measures, which reduce to well known adjusted clustering comparison measures as special cases. Using the theory of generalized IT measures, we are able to propose the following guidelines for using ARI and AMI as external validation indices: ARI should be used when the reference clustering has large equal sized clusters; AMI should be used when the reference clustering is unbalanced and there exist small clusters.
△ Less
Submitted 3 December, 2015;
originally announced December 2015.
-
A Framework to Adjust Dependency Measure Estimates for Chance
Authors:
Simone Romano,
Nguyen Xuan Vinh,
James Bailey,
Karin Verspoor
Abstract:
Estimating the strength of dependency between two variables is fundamental for exploratory analysis and many other applications in data mining. For example: non-linear dependencies between two continuous variables can be explored with the Maximal Information Coefficient (MIC); and categorical variables that are dependent to the target class are selected using Gini gain in random forests. Nonethele…
▽ More
Estimating the strength of dependency between two variables is fundamental for exploratory analysis and many other applications in data mining. For example: non-linear dependencies between two continuous variables can be explored with the Maximal Information Coefficient (MIC); and categorical variables that are dependent to the target class are selected using Gini gain in random forests. Nonetheless, because dependency measures are estimated on finite samples, the interpretability of their quantification and the accuracy when ranking dependencies become challenging. Dependency estimates are not equal to 0 when variables are independent, cannot be compared if computed on different sample size, and they are inflated by chance on variables with more categories. In this paper, we propose a framework to adjust dependency measure estimates on finite samples. Our adjustments, which are simple and applicable to any dependency measure, are helpful in improving interpretability when quantifying dependency and in improving accuracy on the task of ranking dependencies. In particular, we demonstrate that our approach enhances the interpretability of MIC when used as a proxy for the amount of noise between variables, and to gain accuracy when ranking variables during the splitting procedure in random forests.
△ Less
Submitted 20 January, 2016; v1 submitted 27 October, 2015;
originally announced October 2015.
-
Genetic Testing for Complex Diseases: a Simulation Study Perspective
Authors:
Nguyen Xuan Vinh
Abstract:
It is widely recognized nowadays that complex diseases are caused by, amongst the others, multiple genetic factors. The recent advent of genome-wide association study (GWA) has triggered a wave of research aimed at discovering genetic factors underlying common complex diseases. While the number of reported susceptible genetic variants is increasing steadily, the application of such findings into d…
▽ More
It is widely recognized nowadays that complex diseases are caused by, amongst the others, multiple genetic factors. The recent advent of genome-wide association study (GWA) has triggered a wave of research aimed at discovering genetic factors underlying common complex diseases. While the number of reported susceptible genetic variants is increasing steadily, the application of such findings into diseases prognosis for the general population is still unclear, and there are doubts about whether the size of the contribution by such factors is significant. In this respect, some recent simulation-based studies have shed more light to the prospect of genetic tests. In this report, we discuss several aspects of simulation-based studies: their parameters, their assumptions, and the information they provide.
△ Less
Submitted 29 September, 2011; v1 submitted 27 September, 2011;
originally announced September 2011.
