I want to test out various clustering approaches on a dataset. I have the true labels, which can take 4 possible values (1,2,3,4). I want to cluster the observations into these groups based on the other data I have. I can perform the clustering easily enough, but how do I know if the labels given by the algorithm correspond to the label in the original data? For example, the model may assign 2 to all those which are actually label 4, and then when I test for accuracy, all those labelled 2 would be flagged as false, despite being true. Do I need to somehow test out all possible combinations of the final labels, to see which one has the lowest error vs the true labels? Is there an easy way to do this in R?
3
-
$\begingroup$ Clustering is an unsupervised method and is typically implemented when you don't have labels. If you have labels, why not perform classification? $\endgroup$– Ryan VolpiCommented Mar 26, 2021 at 20:23
-
$\begingroup$ @RyanVolpi I'm just playing around with some data so I can practice the methods. Hence, I'd like to be able to benchmark the methods $\endgroup$– Michael21Commented Mar 26, 2021 at 20:27
-
$\begingroup$ I recommend you to read a description Word document of mine, for collection "Compare partitions" on my web-page. There you will find the many agreement measures (some of them mentioned by Lewian's answer) with formulas. $\endgroup$– ttnphnsCommented Mar 26, 2021 at 23:17
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
There are a number of measures to do this, i.e., to compare two clusterings, one of which may be the true one if you have it. The probably most popular one is the adjusted Rand index, computed (not only) by the R-command adjustedRandIndex in package mclust. This is based on comparing for all pairs of observations whether they are in the same cluster in one clustering and also in the other clustering. This would be a positive case, as well as if they are in different clusters in both clusterings, whereas if they are in different clusters in one clustering and in the same cluster in the other one, this would be a negative case for measuring similarity/recovery. This approach does not depend on the numbering/labelling of the clusters, and can also be applied to clusterings that have different numbers of clusters. It is standardised in order to give a value of 1 if clusterings are identical, and 0 if they behave as randomly unrelated clusterings would behave on average, i.e., it can be negative, but 0 and all values below it are very bad results.
References: Adjusted Rand index in:
L. Hubert and P. Arabie (1985) Comparing Partitions, Journal of the Classification, 2, pp. 193-218.
Overview of criteria and some alternatives:
Amigó, E., Gonzalo, J., Artiles, J. et al. (2009) A comparison of extrinsic clustering evaluation metrics based on formal constraints. Inf Retrieval 12, 461–486.
M. Meila (2015) Criteria for Comparing Clusterings. In Hennig, C., Meila, M., Murtagh, F., Rocci, R. (eds.) Handbook of Cluster Analysis. Chapman & Hall/CRC.
answered Mar 26, 2021 at 20:38
-
$\begingroup$ Will the rand index be affected by the choice of label made by the algorithm? $\endgroup$ Commented Mar 26, 2021 at 20:39
-
$\begingroup$ No. As I explained (maybe clearly only in the edit that came after your comment;-). $\endgroup$ Commented Mar 26, 2021 at 20:41
-
$\begingroup$ Yes sorry, I hadn't seen the edit. $\endgroup$ Commented Mar 26, 2021 at 20:42