We have 16 variables which are indices produced by calculations based on ratio (unitless in fact). Some examples of the ranges of our variables are (0.450-0.750), (0.000 - 0.800) and (0.000 - 1.000). Based on this data, we want to apply hierarchical and K-means clustering algorithms. According to the literature, it is recommended to apply standardization before PCA and clustering follows this. In our case, covariance matrix is proposed for PCA but we are not sure we should apply standardization before this process.

If you could help us in this issue, we would be glad.

Thanks in advance for your answers.

  • 2
    $\begingroup$ Clustering algorithms such as k-means and OLS PCA are both sensitive to redundancy in the features and scale, where "scale" refers to the type (e.g., ordinal, interval or ratio) as well as the standard deviations of the features. PCA controls the former issue while some sort of feature transformation is recommended for the latter. This transformation can take many forms such as by the range, ipsative rescaling (e.g., dividing by the maximum value for a series), the IQR, Box-Cox transforms, standardizing to a mean of zero and a std dev of one, etc $\endgroup$ – Mike Hunter Apr 4 '16 at 19:03
  • $\begingroup$ Why are you thinking to do PCA first? You have 16 features that's not at all many to seriously bother with "curse of dimensionality" problem. Doing PCA and dropping some last of the components is potentially fraught with losing information important for the clustering. But in clustering, standardization issue should be considered, of course. $\endgroup$ – ttnphns Apr 5 '16 at 8:13
  • $\begingroup$ @DJohnson What does OLS have to do with PCA? $\endgroup$ – Nick Cox Apr 5 '16 at 8:20
  • $\begingroup$ @NickCox Right! I think we've been down this road. Apologies for forgetting that thread... $\endgroup$ – Mike Hunter Apr 5 '16 at 9:55
  • 1
    $\begingroup$ Thanks for the further detail, but the question is still similar to which car or television or life-partner you should choose out of three (and why those three and no others). We can't see your data and the detail you have given doesn't pin down how well any method will work, especially in relation to your unstated scientific goals. $\endgroup$ – Nick Cox Apr 5 '16 at 20:06

Which literature recommends standardization before PCA on such data? I've never seen this being recommended.

Essentially, PCA is multivariate standardization, so there is some redundancy between first standardizing every attribute on its own, then doing PCA.

As with any form of normalization, it may help, and it may harm. Correlations in your data can be harmful, or helpful. If the variables are corlated because they originate from the same (good) signal, that's great! But if they are correlated because of some root cause that is not helpful, it can be better to reduce the correlation.

  • $\begingroup$ In addition, I forgot to mention that most of our variables are highly correlated and their variances range from 0.002 (min) to 0.036 (max). In that case, which is the best option among the followings: 1. Covariance-based PCA using raw data and then clustering of PCA variables without extra standardization, 2. Just clustering based on raw data (without PCA) 3. Clustering based on standardized data (without PCA). $\endgroup$ – user2067 Apr 5 '16 at 15:15
  • $\begingroup$ There is no automatism that removing correlation improves results. If the correlation is your sinal, then you want to keep it. If it is not interesting, you can want to remove it. Any of the three can be "best" $\endgroup$ – Anony-Mousse -Reinstate Monica Apr 5 '16 at 19:22

Not the answer you're looking for? Browse other questions tagged or ask your own question.