1
$\begingroup$

I've asked this question a few days ago Evaluating HCPC clusters using cluster.stats from fpc library

because I was trying to evaluate the quality of my clusters after I did dimensionality reduction using PCA.

But the x parameter of the silhouette needs the clusters and also needs the distance of the original data matrix if I understand it right https://www.rdocumentation.org/packages/cluster/versions/2.1.0/topics/silhouette

So when I calculate the silhouette for the clusters with PCA I get very low values, for example using just k-means without PCA I get 0.3 avg silhouette and after PCA I get 0.05 avg slhouette.. I think I'm doing something wrong here, it does not make sense to compare the quality of the clusters this way. I did not upload my data, but it is the same for any data, iris for example, does it make sense to run k-means on iris and them run PCA and get the clusters with HCPC and calculate silhoutte in both of them and compare them?

$\endgroup$
4
  • $\begingroup$ It would be indeed helpful if you give the data in your question so we don't speak too in general. As Anony-Mousse remarked, there are more than one potential reason for your finding, but the question itself is an interesting one. $\endgroup$ – ttnphns Aug 3 '19 at 8:07
  • $\begingroup$ You might have done something wrong. You've referred to Iris data. I've just done the analysis of it: clustering (by K-means) of the original dataset (variables standardized) and of the two first PCs of that standardized dataset. In both cases Silhouette index was very similar. About 0.6 for a two-cluster solution and 0.5 for a three-cluster solution; Sihouette fot the PC-based clusterings was even slightly higher. $\endgroup$ – ttnphns Aug 3 '19 at 9:22
  • $\begingroup$ You are right I did wrong I can't find the new matrix to calculate the new distances $\endgroup$ – Ana Aug 3 '19 at 10:52
  • $\begingroup$ Calculate the new distances from the PC scores variates you got from PCA. Use "raw" PC scores, that is, their variances should equal the eigenvalues of PCA. $\endgroup$ – ttnphns Aug 3 '19 at 11:50
2
$\begingroup$

Internal evaluation metrics are not very comparable across different data sets (neither across different projections and other preprocessing). So for fairness, you should indeed use the same distance matrix when computing Silhouette.

At the same time, it is to be expected that the results suffer when not using the original data. PCA rescales the data, making other directions more important. That of course means that a solution found in this rescaled version is not optimized for the original data.

The original data may be the wrong place to compute distances. Then you shouldn't have been clustering there in the first place. First identify the best projection to evaluate distances, then proceed. If distances do not work, Silhouette is meaningless. It always boils down to this question: how to compute meaningful distances. Before you solve this, you can't cluster not evaluate.

Because of this, one may also argue to use Silhouette in the projected space. But as mentioned above, such a projection usually makes the problem easier (by dimensionality reduction) and hence one would expect the Silhouette scores to go up.

Silhouette is still one of the better cases. SSQ values would be completely incomparable!

So either way will not allow you to make a sound statement of whether the result is better with PCA or without. It is always apples with oranges. Thus, I would avoid making any comparative statements! It's just not sound, no matter how you do it.

Instead, I recommend to only draw the following conclusions: - if all Silhouette scores are below 0.2, do not draw any conclusions from them. It's probably the data, not the algorithm, that doesn't work. - if any Silhouette is larger than 0.5 then it is likely a good result - if any Silhouette is larger than 0.7 then it is likely a very good result - if any Silhouette is larger than 0.9 then there is something wrong - if the projected algorithm scores better in Silhouette on the original data, then the projection definitely was a good idea (but the converse does not hold) - if the original clustering scores bettet-or-similar in Silhouette on the PCA-output, then PCA was not necessary (again, the converse does not hold)

$\endgroup$
4
  • $\begingroup$ Thanks a lot! I did wrong because after doing PCA my variables are the dimensions, if I understand it right now I need to calculate silhouette with this new data matrix..I did it with the old one...where is that matrix when you use FactoMineR? I cannot find it...the individuals should be the same but now the columns should be the dimensions and the value in each cell should be totally different $\endgroup$ – Ana Aug 3 '19 at 10:51
  • $\begingroup$ I am saying both have their merits, but also severe issues, and the same algorithm performing "better" on the projected data matrix is a self-fulfilling prophecy... So you really should not compare with either! $\endgroup$ – Has QUIT--Anony-Mousse Aug 3 '19 at 14:33
  • 1
    $\begingroup$ But shouldn't I try to measure the quality of my clusters? I interpreted the clusters obtained with FactoMineR I have clearly different individuals in each of them but now I, $\endgroup$ – Ana Aug 3 '19 at 18:18
  • $\begingroup$ I've no good experiences with internal measures. Look at the data instead of a single score. Usually most clusters are bad, but some are interesting. $\endgroup$ – Has QUIT--Anony-Mousse Aug 3 '19 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.