3
$\begingroup$

Recently I have been working on a scientific paper about clustering, in which I use two extrinsic evaluation metrics to evaluate the clusterings: $B^3 F$ score and $ARI$. The former score is much more informative than the latter and delivers remarkably higher values. On the other hand, $ARI$ seems to fit my dataset in which $k \approx N$ where $k$ is the number of clusters (six on average) and $N$ the data points (20 on average).

However, my supervisory professor suggests omitting $ARI$ because it is insensitive and thus uninformative; for instance, even if I supply the true $k$ to the algorithms it won't change much. Overall $ARI$ produces mediocre results of around $0.1$, which is pretty low.

Upon reading about the limitations of $ARI$ and adjusted measures in general, it was brought to my attention that:

  • The analytical formula of $ARI$ was derived based on the assumption of hyper-geometric model of randomness, which poses some tight constraints that are almost never satisfied by the outputs of a clustering algorithm [1];
  • Moreover, Meila [2] explained that all adjusted indices, including ARI, are non-local, which means a variation in one of the clusters would be considered differently depending on how the remaining clusters are formed.

So could you please further elaborate on the situations in which the usage of $ARI$ isn't advocated?


[1] http://www.jmlr.org/papers/volume11/vinh10a/vinh10a.pdf [2] https://www.stat.washington.edu/mmp/Papers/icml05-compare-axioms.pdf

$\endgroup$

1 Answer 1

6
$\begingroup$

First of all, ARI is one of the standard measures used everywhere. So if you omit it, there is a good chance reviewers will demand that you add it, and reject your paper. B³ is a fairly exotic measure that I have not seen anybody actually use.

It is meaningless if the values of measure A are higher than those of measure B. Because that is apples and oranges. Conversely, some fairly simple transformations can affect these values. For example. The Rand index (RI) will always be higher than ARI, despite them measuring the same quantity, because ARI take the RI relative to an expected value. So B³>ARI is a useless observation, you must never compare different measures. Do you actually observe different rankings? I.e. give two results R1, R2, do you observe $B³(R1)\gg B³(R2)$ but $ARI(R1)\ll ARI(R2)$ on the same data set?

Which essentially brings us to a point why you should pay attention to ARI. If your result has an ARI close to 0 (such as 0.1), then it means your result will be almost as good if you randomly permute all labels. But then how can it be any good? Does B³ do any such adjustment? What is the B³ of a random result?

It is correct that ARI is not perfect. But are the other measures actually better there? I doubt so. Yes, the adjustment of ARI uses a fairly simple assumption for adjustment (that breaks down for constant labels), but it serves the purpose of making the Rand index more interpretable well. For some uncommon special cases, it may be desirable to use different adjustments. In such cases, I would then suggest to use the minimum of all adjustments. So your score can only become worse.

Non-locality is an unfortunate property but not crucial for most users. It this mostly is relevant for hierarchical approaches: while you can locally evaluate Rand to check if a split yields better results, you cannot find the best agreement with respect to ARI that easily. The reasonable approach in this situation is of course to maximize the Rand index, and then adjust this final result.

In conclusion, I suggest that you:

  1. Always give the standard ARI index. A low ARI does indicate a poor result.
  2. If you want, add additional adjustments with other null models.
  3. Give the NMI, because that is the other standard measure (it also has issues; in particular a high NMI does not guarantee a good result, because a random result can score high). Be explicit about which version of NMI you use!
  4. Give the AMI, the adjusted mutual information. This is a similar adjustment that aims at normalizing a random MI to be 0.
  5. Include trivial baselines, such as random permutation of the true labels, all objects labeled 0, random subsampling of k "centers" and assigning each to its nearest "center" etc. If you can't substantially beat these baselines, your method is not good (and you will be surprised by how common this is!)
$\endgroup$
3
  • $\begingroup$ Thank you for taking the time to respond and sorry for the late comment. Just some notes: I don't know if by B² you mean B³F [e-spacio.uned.es/fez/eserv/bibliuned:DptoLSI-ETSI-MA2VICMR-1090/…, which I actually adopt. Of course it's senseless to compare two different metrics, I didn't mean to do so, but to generally hint that the quality of results vary considerably across the metrics. In the context of the question, you're saying that $ARI$ shouldn't be used within a hierarchical clustering algorithm, but it should be used elsewhere, am I right? $\endgroup$
    – Rafs
    Commented Oct 26, 2019 at 8:43
  • $\begingroup$ Sorry, I meant B³. It's very exotic, and I have not seen any independent confirmation of it's usefulness. As far as I remember, it - just like all the other measures - is only defined on disjoint partitions, not on hierarchies. So it cannot be correctly used. $\endgroup$ Commented Oct 26, 2019 at 10:53
  • 1
    $\begingroup$ What you could do for B³ is to compute the score distribution of random labelings. Of course that won't be as good as an anlytical solution as done in ARI. $\endgroup$ Commented Oct 26, 2019 at 10:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.