7
$\begingroup$

I have read around a lot and tried different ways to carry out my cluster analysis. In the first case, I have carried out a hierarchical cluster analysis on my raw data (200 watersheds and 16 variables) in matlab and mapped the clusters.

My second attempt incorporated carrying out a Principal Components Analysis on the raw data and then using the scores as inputs to my hierarchical clustering.

The clusters produced in both cases are the exact same, which I did not expect. Can anyone explain to me why they are the same?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
14
$\begingroup$

This is because PCA scores are simply original data in a rotated coordinate frame.

Below on the left I show some example 2D data (100 points in 2D) and on the right the corresponding PCA scores. The data cloud simply gets rotated clockwise by approximately 45 degrees.

Toy dataset: original data and PC scores

If it is not completely clear to you how one gets from the first subplot to the second one or why PCA amounts to rotation, take a look at our very informative thread Making sense of principal component analysis, eigenvectors & eigenvalues. In my answer there I am using exactly the same toy dataset as displayed here. Some other answers are very much worth reading too.

Now, to your question.

Clustering methods are usually based on Euclidean distances between points. The points that lie close to each other get clustered together; the ones that are far away get assigned to different clusters. As you can see above, all distances between all points stay exactly the same after PCA.

Hence the identical clustering results. Here are both representations clustered with k-means with $k=3$:

K-means on the original data and on the PCA-rotated data

As you see, the clustering results are identical.

Can PCA make any difference at all?

Yes. One can use it in two ways:

  1. Standardize all scores to unit variance; or
  2. Use only a subset of principal components, usually the ones that explain the most variance.

Here is how it looks like in the same toy example. On the left I am using standardized scores (note how different the clusters become), on the right I am using only PC1.

K-means on standardized PC scores and K-means on PC1

Improve this answer
$\endgroup$
16
  • 1
    $\begingroup$ @Amoeba That is an incredibly informative answer. Thank you. I have one follow-on question to clarify exactly what you have said. PCA literature suggests that the data be standardised (if the variables are measured in different units..which mine are) before PCA. In the final part of your answer you state that the scores should be standardised pre-clustering..Is this the case, even if the raw data was standardised before PCA? $\endgroup$
    – matlab_newby
    Commented Aug 18, 2016 at 8:52
  • 2
    $\begingroup$ @matlab_newby I am glad it was helpful. I did not say that the scores should be standardized before clustering; I said that if you standardize them then your clustering results will (or at least may) be different. To your question: if you want to standardize scores then yes, you should do it even if the original variables were standardized prior to PCA. Look at my toy data: variance of $x$ and $y$ is around the same, but the variance of PC1 and PC2 is very different. So a standardized dataset can produce scores that have very different variances. $\endgroup$
    – amoeba
    Commented Aug 18, 2016 at 12:46
  • $\begingroup$ So, if one were to take the second route you mention above: ie. to 'Use only a subset of principal components, usually the ones that explain the most variance'...that you are referring to the loadings/coefficients of the first 'n' PCs (n being the number of newly derived PC 'variables' that explain the most variance)...and that this subset of PCs is used as input to clustering (CA)? If so, the CA would be clustering PC coefficients, as opposed to observations (obs), which would make the interpretation of the clusters (in relation to the original obs) very difficult? $\endgroup$
    – matlab_newby
    Commented Aug 18, 2016 at 14:26
  • $\begingroup$ @matlab_newby No no no! You always use PC scores! I meant taking scores corresponding to the first $n$ PCs. $\endgroup$
    – amoeba
    Commented Aug 18, 2016 at 14:27
  • $\begingroup$ Ok. I get it now. The second route only refers to using a subset of PC scores for input to the CA. That allows for clustering on the observations and then you use the loadings to ascertain which variables are most dominant on the chosen PCs. And if one were to use all PC scores, then it would be necessary to 'Standardize all scores to unit variance' as in option 1 above. $\endgroup$
    – matlab_newby
    Commented Aug 18, 2016 at 15:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.