1
$\begingroup$

Can I use accuracy, sensitivity, specificitity, positive predictive value, negative predictive value when i'm doing unsupervised classification (clustering) or do I have to limit my clustering evaluation to average silhouette width?

if you wonder how come I'm talking about accuracy while working on clustering, it's just because i have the actual real labels of all my subjects (healthy and sick) and I just want to see how well the clustering predicted the real labels with a specific variable.

And please, if you have other relevant evaluation criteria do not hesitate!

$\endgroup$
2
$\begingroup$

Let the classes have labels "healthy" and "sick".

If you run kmeans, you get clusters labeled 0,1,2,3,4

How would you compute any of these measures?

There are plenty of evaluation measures for clustering. They are related to classification measures, but not the same, for a reason... Use the clustering measures for cluster evaluation and the classification evaluation measures for classification evaluation.

The two most popular cluster evaluation measures seem to be ARI and NMI.

$\endgroup$
  • $\begingroup$ Could you provide references or links for the measures you recommend? $\endgroup$ – Tim May 4 '18 at 9:48
  • $\begingroup$ @Anony-Mousse If I run kmeans with k=2 so the predicted clusters labels are: 0 and 1. Then, I just decide that "healthy"= 0 and "sick"=1, then I have the confusion matrix (predicted clusters VS real clusters). and with the confusion matrix I can have the true positives, true negatives, false positives, false negatives and so I can calculate the accuracy... No? $\endgroup$ – learneRS May 4 '18 at 11:15
  • $\begingroup$ What if they are the other way around? Or you need to use k=5 because some clusters are just single nose points? Use ARI and NMI as in Wikipedia and any good clustering tutorial. $\endgroup$ – Anony-Mousse May 4 '18 at 23:41
0
$\begingroup$

There is a major underlying hypothesis in your process. You are indeed assuming that the clustering returns two distinct groups, and each of them should corresponds to your labels, which may not be the case.

Keep in mind that clustering is a label-free process. The two groups and especially their labels may vary depending on the initialization. Your process needs to identify which one represents (supposedly) healthy subjects, and which one should represents sick subjects, regardless of the label returned by the clustering.

Assuming that you have tackled this issue and identified who is who (say for instance, by looking at the group with highest proportion of truly sick subjects), you can indeed measure the accuracy of the model that you have created with the metrics you have mentioned. All metrics that can be applied to binary classification are relevant here. The biggest issue relies in the technique used to assign the right label to the groups created by your clustering.

$\endgroup$
  • 2
    $\begingroup$ Keep in mind, though: 1. identifying which cluster correponds to which class basically turns your cluster analysis into a classification. 2. Measuring accuracy etc. on the initially clustered data set corresponds to a training error. It is intrinsically biased (in an optimistic way): when correlating clusters with labels, you already exclude the possibility that a classifier can be totally off (worse than guessing). You need to assess the naive guessing accuracy etc. for your method of assigning labels to clusters. $\endgroup$ – cbeleites May 3 '18 at 15:29
  • $\begingroup$ thank you also to @cbeleites What do you suggest? $\endgroup$ – learneRS May 3 '18 at 15:50
  • $\begingroup$ @cbeleites Totally agreed. I was making the assumption that the initial method used to build the model was based on clustering, and that they had a relevant method to turn it into classification (which is in any case necessary when predicting something else than the training set). The whole point of my post was to underline how critical is the method used for clustering-classification mapping. $\endgroup$ – AshOfFire May 3 '18 at 16:02
  • $\begingroup$ So as conclusion I can't use accuracy in this case I have to use the classical measures for clustering quality: Rand Index, silhouette width etc...? $\endgroup$ – learneRS May 3 '18 at 16:11
  • $\begingroup$ It depends if your final goal is purely descriptive (e.g. clustering to discover new patterns) or predictive (e.g. turn your clustering into classification). In the first case, accuracy is irrelevant. In the second case, it is relevant to keep it. $\endgroup$ – AshOfFire May 3 '18 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.