1
$\begingroup$

I'm currently confused on choosing the method for evaluating different clustering techniques. From this paper, they followed the pipeline: use Hungarian assignment for matching the cluster with true label, then calculate F1, Precision, Recall like classification problem. But other papers mentioned about Adjusted Rand Index (ARI).

I have tried both method, and I found that most of the case, the ARI give higher score than the former method. Especially, there's one case that ARI gives 0.96 score, which is close to perfect matching, but F1 gives only 0.50, which is very bad clustering.

I wonder that is there any 'official' comparison between using the two methods, and which one is more reliable/widely acknowledged?

$\endgroup$

2 Answers 2

3
$\begingroup$

You must not and cannot compare scores from evaluation method A to scores from evaluation method B.

In particular, ARI is adjusted for chance - a random result will score 0 - as oppose to the regular Rand index; which in turn is the accuracy of the pairs. But precision/recall/f1 on a balanced k-class problem will be around 1/k for a random result. So a higher score in measure A does not mean the result is better than a lower score with measure B!

The standard measures for evaluating clusterings with respect to existing labels (the usual caveats apply) in literature are ARI and NMI (usually in the sqrt version, but probably AMI would be the better choice here, too).

Matching with the Hungarian algorithm is possible, but even more assumes that there "must" be a 1:1 correspondence of clusters and classes - an assumption that has repeatedly proven wrong, classes can have substructures that are not labeled. It may be reasonable if the application requires a precisely defined number of clusters (e.g., in signal transmission, when you know there must be 10 different signals on the wire; there won't be 11).

Furthermore, ARI and NMI are more robust to changes in the labeling. Relabeling a single point could mean the Hungarian matching algorithm produces a very different cluster mapping. So I'd argue the ARI and NMI results likely are more reliable.

$\endgroup$
2
  • $\begingroup$ thank you for such a great answer. Could you please suggest some journal articles related to Evaluation of Clustering? I just found this one: nlp.stanford.edu/IR-book/html/htmledition/… , I think I need more reference to convince my supervisor $\endgroup$ Dec 12, 2018 at 4:46
  • $\begingroup$ Look outside the IR community. Because that is where clustering research happens. IR is a bit different, purity for example only makes sense with partial results, as in IR. As a starting point, use this paper with 1000+ citations: Vinh, Nguyen Xuan, Julien Epps, and James Bailey. "Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance." Journal of Machine Learning Research 11.Oct (2010): 2837-2854. $\endgroup$ Dec 12, 2018 at 6:21
1
$\begingroup$

The two proposed methods are used for different purposes. The first method tries to match the data first and then use standard classification metrics, so it's about predictive accuracies. The ARI measures similarity between clusters from different procedures.

If the goal is to compare different clustering procedures then I would use the ARI, if however the goal is to find which clustering performs better in a classification task then I would use the first approach (not exactly, but something similar).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.