I want to test out various clustering approaches on a dataset. I have the true labels, which can take 4 possible values (1,2,3,4). I want to cluster the observations into these groups based on the other data I have. I can perform the clustering easily enough, but how do I know if the labels given by the algorithm correspond to the label in the original data? For example, the model may assign 2 to all those which are actually label 4, and then when I test for accuracy, all those labelled 2 would be flagged as false, despite being true. Do I need to somehow test out all possible combinations of the final labels, to see which one has the lowest error vs the true labels? Is there an easy way to do this in R?
There are a number of measures to do this, i.e., to compare two clusterings, one of which may be the true one if you have it. The probably most popular one is the adjusted Rand index, computed (not only) by the R-command adjustedRandIndex in package mclust. This is based on comparing for all pairs of observations whether they are in the same cluster in one clustering and also in the other clustering. This would be a positive case, as well as if they are in different clusters in both clusterings, whereas if they are in different clusters in one clustering and in the same cluster in the other one, this would be a negative case for measuring similarity/recovery. This approach does not depend on the numbering/labelling of the clusters, and can also be applied to clusterings that have different numbers of clusters. It is standardised in order to give a value of 1 if clusterings are identical, and 0 if they behave as randomly unrelated clusterings would behave on average, i.e., it can be negative, but 0 and all values below it are very bad results.
References: Adjusted Rand index in:
L. Hubert and P. Arabie (1985) Comparing Partitions, Journal of the Classification, 2, pp. 193-218.
Overview of criteria and some alternatives:
Amigó, E., Gonzalo, J., Artiles, J. et al. (2009) A comparison of extrinsic clustering evaluation metrics based on formal constraints. Inf Retrieval 12, 461–486.
M. Meila (2015) Criteria for Comparing Clusterings. In Hennig, C., Meila, M., Murtagh, F., Rocci, R. (eds.) Handbook of Cluster Analysis. Chapman & Hall/CRC.