12
$\begingroup$

I am trying to evaluate clustering performance. I was reading the skiscit-learn documentation on metrics. I do not understand the difference between ARI and AMI. It seems to me that they do the same thing in two different ways.

Citing from the documentation:

Given the knowledge of the ground truth class assignments labels_true and our clustering algorithm assignments of the same samples labels_pred, the adjusted Rand index is a function that measures the similarity of the two assignments, ignoring permutations and with chance normalization.

vs

Given the knowledge of the ground truth class assignments labels_true and our clustering algorithm assignments of the same samples labels_pred, the Mutual Information is a function that measures the agreement of the two assignments, ignoring permutations ... AMI was proposed more recently and is normalized against chance.

Should I use both of them in my clustering evaluation or would this be redundant?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Mr. Rand is not random. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Feb 7, 2017 at 20:30

2 Answers 2

Reset to default
23
$\begingroup$

Short answer

  • Use ARI when the ground truth clustering has large equal sized clusters
  • Use AMI when the ground truth clustering is unbalanced and there exist small clusters

Longer answer

I worked on this topic. Reference: Adjusting for Chance Clustering Comparison Measures

A one-line summary of the paper is: AMI is high when there are pure clusters in the clustering solution.

Let's have a look at an example. We have a reference clustering V consisting of 4 equal size clusters. Each cluster is of size 25. Then we have two clustering solutions:

  • U1 that has pure clusters (many zeros in the contingency table)
  • U2 that has impure clusters

enter image description here

AMI will choose U1 and ARI will choose U2.

Eventually:

  • U1 is unbalanced. Unbalanced clusters have more chances to present pure clusters. AMI is biased towards unbalanced clustering solutions
  • U2 is balanced. ARI is biased towards balanced clustering solutions.

If we are using external validity indices such as AMI and ARI, we are aiming at matching the reference clustering with our clustering solution. This is why the recommendation at the top: AMI when the reference clustering is unbalanced, and ARI when the reference clustering is balanced. We do this mainly due to the biases in both measures.

Also, when we have an unbalanced reference clustering with small clusters, we are even more interested in generating pure small clusters in the solution. We want to identify precisely the small clusters from the reference. Even a single mismatched data point can have a relatively higher impact.

Other than the recommendations above, we could use AMI when we are interested in having pure clusters in the solution.

Experiment

Here I sketched an experiment where P generates solutions U which are balanced when P=1 and unbalanced when P=0. You can play with the notebook here.

enter image description here

Improve this answer
$\endgroup$
14
  • 1
    $\begingroup$ pastebin.com/raw/WHvTxbLm This is one of the cases that I do not understand: Better AMI does not mean better ARI and vice-versa. Is there any reason why I would trust the relative improvement of one or the other. I am not sure which metric to look at in order to improve my results (from the paper you linked, I sense it should be AMI given my class distribution but I am still confused). $\endgroup$
    – ryuzakinho
    Commented Nov 15, 2018 at 12:39
  • 1
    $\begingroup$ In you case, HDBSCAN result shows a very big cluster and many small ones which is by definition an unbalanced solution. Hence AMI is bigger with DBSCAN. Your ground truth is more balanced than that solution. Therefore, I would use ARI to choose the solution here. This said, it seems that the clustering solutions you obtained are not that good. Maybe it is because you have many clusters. Could you reduce the number of clusters that you want? Or do you have features to take into account rather than using a purely distance based clustering? $\endgroup$
    – Simone
    Commented Nov 15, 2018 at 15:24
  • 1
    $\begingroup$ After more qualitative testing, it turns out that AMI was more reliable for my use case. Indeed, AMI said that the HDBSCAN was better, and I found it better indeed. Although I had one big noise cluster, the other clusters were purer than the KMEANS clusters. $\endgroup$
    – ryuzakinho
    Commented Nov 30, 2018 at 10:48
  • 1
    $\begingroup$ Neither ARI nor AMI understands noise or hierarchies. You should not use them on a hierarchical clustering result with noise. $\endgroup$
    – Erich Schubert
    Commented Mar 17, 2022 at 17:02
  • 1
    $\begingroup$ There are some density-based measures that might be more suitable. But they key question is: what measure is relevant for your problem. There is no use in measuring "quality" with a measure that measures the wrong thing for you. But is there any use for label based evaluation besides ivory tower paper-writing? $\endgroup$
    – Erich Schubert
    Commented Mar 27, 2022 at 12:04
4
$\begingroup$

They are two out of a dozen that all try to compare clusterings.

But they are not equivalent. They use different theory.

Sometimes, ARI may prefer one result and the AMI another. But often they agree in preference (not in the numbers).

Improve this answer
$\endgroup$
3
  • $\begingroup$ What do you mean with: " they agree in preference (not in the numbers)? " $\endgroup$
    – al27091
    Commented Feb 8, 2017 at 7:35
  • $\begingroup$ When you compare multiple results. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Feb 8, 2017 at 20:36
  • $\begingroup$ @al27091: Has QUIT--Anony-Mousse means that when you compare partitions of a set of algorithms' with a reference partition using both ARI and AMI, then most likely both ARI and AMI will rank the algorithms' result in the same way. However, the numbers of ARI and AMI will be different. $\endgroup$
    – Make42
    Commented Jun 8, 2021 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.