1
$\begingroup$

I try to do an (unsupervised) clustering with sklearn, Python, by different algorithms (hierarchical, distance based, density based etc. ones). The data in question has few hundred original features and I do experiments with different dimension reduction algorithms before the actual clustering (PCA, IncrementalPCA, LocallyLinearEmbedding, Isomap, FactorAnalysis).

The strange tendency I observe is that the fewer component I keep with dimension reductions (more-less irrespective to the kind of the algorithm) the better result I get (by silhouette score). This tendency is true to the extreme: the best result seems to be the one with only 2 compressed final dimension.

This seems absurd, my question is what is the explanation here? I could imagine that either silhouette score is misleading in the evaluation (but then what to chose?), or that I may still happen to have a few key original explanatory variable which is enough to be kept?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ I'm not sure that this is generally answerable, because it will depend partly on the specifics of your dataset. And it's not even necessary for the first few PCs to be the most important ones. But you might be interested in our threads on the curse of dimensionality $\endgroup$
    – mkt
    Aug 26, 2022 at 5:34
  • $\begingroup$ Did you try to search the site for similar questions? There have been not one such. $\endgroup$
    – ttnphns
    Aug 28, 2022 at 0:48
  • $\begingroup$ Did you check how Silhouette statistic react to dimensionality in random datasets with no clusters? $\endgroup$
    – ttnphns
    Aug 28, 2022 at 0:55
  • $\begingroup$ Check e.g. this stats.stackexchange.com/q/222675/3277 $\endgroup$
    – ttnphns
    Aug 28, 2022 at 1:00

1 Answer 1

Reset to default
3
$\begingroup$

The problem with clustering after dimensionality reduction is that the clustering can completely change by the projection. See e.g. this example:

enter image description here

Here, in higher dimensions, you have five perfect clusters and after PCA projection (to the red line) you end up with just one cluster. And even though the clustering after dimensionality reduction is nowhere near the proper clustering, the silhouette score can still be good, maybe even better than for the original data.

But clearly, this is not only a problem with the silhouette score, this follows from the projection which can severely alter the grouping of your data. So comparing any scores for clusterings after different dimensionality reductions is often not very informative.

Clustering metrics, especially in higher dimensions, are always problematic. None of them have ever worked for me except for toy examples. In my experience, methods like t-SNE and UMAP still work best.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ +1 especially for noting it that far not every clustering validity measure is insensitive to changing the dimensionality. $\endgroup$
    – ttnphns
    Aug 28, 2022 at 0:53
  • $\begingroup$ Your explanation seems to suggest that applying dimensionality reduction before clustering is unwise in general, is that correct? If so, how about addressing the issues with the original data (high dims, sparsity, etc.) aren't they not bigger problem? $\endgroup$
    – Fredrik
    Aug 30, 2022 at 6:27
  • 1
    $\begingroup$ @Fredrik Clustering at high dimensions is often a lost cause. And dimensionality reduction before clustering, in particular linear projections, is not helping. You lose too much information. As I said, the best you can do, in my experience, is to use nonlinear methods like t-SNE or UMAP, which are trying to preserve the grouping structure. $\endgroup$
    – frank
    Aug 30, 2022 at 7:04
  • $\begingroup$ @frank, how do you figure out whether a given clustering method "works" or not? I think this is the ultimate question (although a bit different topic, I acknowledge). $\endgroup$
    – Fredrik
    Aug 30, 2022 at 7:15
  • $\begingroup$ @Fredrik I agree. The problem is, there is no "definition" of a "cluster". See also this. In a way, each clustering algorithm is a new definition of a cluster. $\endgroup$
    – frank
    Aug 30, 2022 at 16:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.