4
$\begingroup$

I'm performing clustering on a labeled dataset. I would like to check the quality of clustering. Is there a well accepted way of doing that?

So basically I would like perform some classification-like procedure, where I would determine the quality of the clustering, since I already have the labeled dataset. Does that make sense and how to do it?

$\endgroup$
1
  • 1
    $\begingroup$ Sure. Checking whether clustering has classified well according to some preexistent labels, that is, whether the clustering supports (= is supported by) some outer classification, is called external-criterion clustering validation. Wikipedia on cluster analysis mentions some approaches. $\endgroup$ – ttnphns Jul 5 '15 at 23:02
4
$\begingroup$

There are many many measures that can be used on labeled data.

For example, if you run k-means wiht $k=3$ on the mouse data set:

k-means on Mouse

you get the following evaluation result (using ELKI):

Evaluation result

Clearly, it did not work very well. If you know this toy data set, k-means just doesn't work well on it, because the clusters have too different size.

These are external evaluation measures. They work well if the labels correspond to clusters. If you are using classification data, the labels may not at all correspond to clusters; but some classes may form one big cluster, or a class may split into multiple clusters. There may also be outliers. They work well on synthetic data, but real data just never has such labels already.

So while such measures are a nice thing for experimenting, they have big issues... IMHO, their results can be totally misleading. A clustering algorithm that works perfectly well may score really bad on such a measure, if the labels do not correspond to the data clustering structure.

Clustering just is not classification. It's rather orthogonal.

$\endgroup$
4
  • $\begingroup$ I haven't quite understand this: They work well if the labels correspond to clusters. It may sound as if the correspondence is an assumption under which one may apply an external evaluation measure. But the whole task of the evaluation really is to try to discover that correspondence - the one that maximizes the measure - and then to see how big is that maximized measure. $\endgroup$ – ttnphns Jul 6 '15 at 10:09
  • $\begingroup$ Labels could literally be anything. For example prime numbers. But you cannot expect a clustering algorithm to pull out prime numbers. Labels could store a property of the data that does not reflect any form of clustering. It does not help choosing an appropriate clustering algorithm then. $\endgroup$ – Has QUIT--Anony-Mousse Jul 6 '15 at 12:52
  • $\begingroup$ Anony, could you direct me to the place where ELKI discuss / show their formula of the Gini measure of the labeled validation? $\endgroup$ – ttnphns Mar 11 '19 at 18:08
  • $\begingroup$ No idea if this works: ping @ErichSchubert $\endgroup$ – Has QUIT--Anony-Mousse Mar 12 '19 at 0:30
4
$\begingroup$

If your data is labeled that is the true classification of your data set. Then you can apply any of the know clustering methods (hierarchical, kmeans or model-based clustering) and use the adjustedRandIndex. This is a function in R in the mclust package. Adjusted Rand Index indicates how similars the clusters are, and when the value is 1 it means that they are identical. Hence you will compare the tue clustering with the result of the different clustering methods. There is another function in R that does approx the same thing it is call Error Rate or something like this and it computes the proportion of the truly identified clusters from the clustering method you have chosen.

$\endgroup$
2
  • $\begingroup$ Thanks for your suggestion, Thomas. I'm actually not using R, but experimenting using Weka, but I guess that I could implement something like that. I will not close the question yet, to see if there are any other suggestions. $\endgroup$ – Kobe-Wan Kenobi Jul 5 '15 at 21:09
  • $\begingroup$ If you are using Weka, you may want to have a look at ELKI instead. It has many more clustering algorithms and will automatically run various quality measures (including ARI) on it if you have labeled data. $\endgroup$ – Has QUIT--Anony-Mousse Jul 6 '15 at 8:08
3
$\begingroup$

Clustering is usually used for unsupervised classification - that is, when you are trying to discover groups that might exist but that you don't know about.

If you want to classify units into known groups, you could look at multinomial logistic regression or at classification trees and related methods.

$\endgroup$
3
  • 3
    $\begingroup$ Thank you for your answer, Sir. I'm aware of that, but I'm aiming towards something like Rand Index that Thomas mentioned. I have a labeled corpus, but I would like to cluster it like it's not labeled and to find the "accuracy of the clustering", that is, I would like to compare different clustering algorithms (or only distance measures, perhaps) to decide which seems to be the best clustering algorithm for the corpus. I now that by definition clustering is unsupervised and I'm using it in that way, I just want to make a knowledge of how good the clusters are. $\endgroup$ – Kobe-Wan Kenobi Jul 5 '15 at 21:08
  • $\begingroup$ @pera, you are right at what you are planning. But be cautious. "Best" clustering method or distance might be best by chance of your specific sample. Cross-validation or analysis by subsamples should be performed to make sure that "best" is really stably the best. And another notion: there is a better idea first to investigate the distribution in the classes of your population and then just decide what clustering method could be the best for it. For example, if classes are approximately normal and spherical, you may prefer K-means or Wards hierarchical method. $\endgroup$ – ttnphns Jul 5 '15 at 23:17
  • $\begingroup$ Thank you for the explanation. How would you determine that in the case of big multidimensional dataset? For example, if I'm clustering documents crawled from Wikipedia, how could I derwrmine what to use? $\endgroup$ – Kobe-Wan Kenobi Jul 6 '15 at 7:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.