9
$\begingroup$

I am trying to evaluate clustering performance. I was reading the skiscit-learn documentation on metrics. I do not understand the difference between ARI and AMI. It seems to me that they do the same thing in two different ways.

Citing from the documentation:

Given the knowledge of the ground truth class assignments labels_true and our clustering algorithm assignments of the same samples labels_pred, the adjusted Rand index is a function that measures the similarity of the two assignments, ignoring permutations and with chance normalization.

vs

Given the knowledge of the ground truth class assignments labels_true and our clustering algorithm assignments of the same samples labels_pred, the Mutual Information is a function that measures the agreement of the two assignments, ignoring permutations ... AMI was proposed more recently and is normalized against chance.

Should I use both of them in my clustering evaluation or would this be redundant?

$\endgroup$
  • $\begingroup$ Mr. Rand is not random. $\endgroup$ – Anony-Mousse Feb 7 '17 at 20:30
2
$\begingroup$

They are two out of a dozen that all try to compare clusterings.

But they are not equivalent. They use different theory.

Sometimes, ARI may prefer one result and the AMI another. But often they agree in preference (not in the numbers).

$\endgroup$
  • $\begingroup$ What do you mean with: " they agree in preference (not in the numbers)? " $\endgroup$ – al27091 Feb 8 '17 at 7:35
  • $\begingroup$ When you compare multiple results. $\endgroup$ – Anony-Mousse Feb 8 '17 at 20:36
7
$\begingroup$

The rule of thumb is:

  • Use ARI when the ground truth clustering has large equal sized clusters
  • Usa AMI when the ground truth clustering is unbalanced and there exist small clusters

I worked on this topic. Reference: Adjusting for Chance Clustering Comparison Measures

$\endgroup$
  • $\begingroup$ I applied HDBSCAN and KMeans on some of my datasets with the right number of clusters for KMeans and the correct min cluster size for HDBSCAN. My problem is that a progression in AMI doesn't correlate with a progression in ARI. I get a mean of 0.3 and 0.35 in AMI which is low. I get ARI results close to 0: 0.07 and 0.01 mean respectively. Even in the cases where I obtained better AMI with HDBSCAN, my ARI scores were very close to 0 i.e HDBSCAN produces lower ARI than KMeans even in the cases where the AMI is higher. $\endgroup$ – ryuzakinho Nov 15 '18 at 10:43
  • $\begingroup$ What type of clustering results do mean 0.3 and 0.35 for AMI correspond to? $\endgroup$ – Simone Nov 15 '18 at 10:59
  • 1
    $\begingroup$ pastebin.com/raw/WHvTxbLm This is one of the cases that I do not understand: Better AMI does not mean better ARI and vice-versa. Is there any reason why I would trust the relative improvement of one or the other. I am not sure which metric to look at in order to improve my results (from the paper you linked, I sense it should be AMI given my class distribution but I am still confused). $\endgroup$ – ryuzakinho Nov 15 '18 at 12:39
  • 1
    $\begingroup$ In you case, HDBSCAN result shows a very big cluster and many small ones which is by definition an unbalanced solution. Hence AMI is bigger with DBSCAN. Your ground truth is more balanced than that solution. Therefore, I would use ARI to choose the solution here. This said, it seems that the clustering solutions you obtained are not that good. Maybe it is because you have many clusters. Could you reduce the number of clusters that you want? Or do you have features to take into account rather than using a purely distance based clustering? $\endgroup$ – Simone Nov 15 '18 at 15:24
  • 1
    $\begingroup$ After more qualitative testing, it turns out that AMI was more reliable for my use case. Indeed, AMI said that the HDBSCAN was better, and I found it better indeed. Although I had one big noise cluster, the other clusters were purer than the KMEANS clusters. $\endgroup$ – ryuzakinho Nov 30 '18 at 10:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.