3
$\begingroup$

I'm working on comparing multiple clustering algorithms to each other using the adjusted Rand index for a given dataset. We have a gold standard that we'd like to compare the obtained clustering assignments against. My main question is:

Is it common place to use cross-validation to compare adjusted rand index values?

I can't seem to find a good answer to my question in the literature. The other problem is that for some clustering algorithms I'm using, have really no good way of computing the clustering assignment for data that is removed. I'm thinking of hierarchal clustering and spectral clustering.

In place of running cross-validation, I'm simply rerunning the clustering techniques 10 times and then computing t-test to determine if the difference is statistically significant. My clustering analysis days have me thinking that this is going to be a problem.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ In an algorithm that depends on randomized starting points (like many clustering algorithms) it is always appropriate to re-run the algorithm several times with different starts $\endgroup$
    – shadowtalker
    Commented Mar 10, 2015 at 13:38

1 Answer 1

Reset to default
1
$\begingroup$

I'm not terribly familiar with the Rand Index & haven't used it, but why would it be inappropriate, even if seldom cited? This conference paper by Santos & Embrechts discusses adjusted Rand index for supervised classification & uses k-fold cross-validation. http://www.dema2.isep.ipp.pt/~jms/pubs/LNCS5769.pdf

Additionally, the following thesis by O'Reilly discusses cross-validation for ARI: https://dspace.lib.uoguelph.ca/xmlui/bitstream/handle/10214/3911/mythesis.pdf?sequence=1

Your t-test idea may not be too far off. In addition to citing some other methods in their introduction, Tibshirani & Walther cite an ANOVA procedure as a method for determining appropriate clusters. (Tibshirani & Walther. 2005. Journal of Computational and Graphical Statistics, Volume 14, Number 3, Pages 511–528).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.