8
$\begingroup$

After doing PCA, the first component describes the largest part of variability. This is important e.g. in study of body measurements where it is commonly known (Jolliffe, 2002) that PC1 axis captures size variation. My question is whether PCA scores after varimax rotation retain the same properties or are they different as mentioned in this topic?

Since I need PCA scores for further statistical analyses I am wondering if varimax is needed and does it in fact disrupt the representation of real sample variability so that individual scores on rotated axes are uninformative or lead to miss-interpretation of reality?

Also could someone suggest some other references on this topic?

Workflows in R:

  1. PCA (FactoMineR or prcomp) -> Extract individual scores -> Enter scores in the lm
  2. PCA (FactoMiner or prcomp) -> Varimax on loadings matrix -> calculate the individual scores -> enter scores in the lm
  3. FA (psych, varimax and pca extraction method) -> extract individual scores -> Enter scores in the lm

Now, without rotation (1.) percentages of explained variability are i.e. 29.32, 5.6, 3.2, on the first three axes. 2. and 3. solutions yield similar percentages on the first three factors i.e. 12.2, 12.1, 8.2. Off course 1. solution tends to push all high variable loadings on the first axis, while 2. and 3. tend to distribute loadings between axes (which is the reason for rotation). I wanted to know if these three workflows are essential the same since individual scores are different on rotated vs. unrotated axes?

$\endgroup$
  • $\begingroup$ The purpose of rotation is to share out the variability amongst the components, so that's not true any more. I'm not sure what you mean by "disrupt the representation of real sample variability" but rotation does not change the proportion of variance explained in each item by the components. $\endgroup$ – Jeremy Miles Apr 14 '13 at 16:49
  • 2
    $\begingroup$ Rotation is performed solely for the sake of interpretation of the components. Interpretation is needed solely in the context of "latent variable model", i.e. when you treat a component as a factor, = you take PCA as Factor analysis [a permissible albeit controversial decision]. Is that your case? $\endgroup$ – ttnphns Apr 14 '13 at 17:07
  • 1
    $\begingroup$ As was correctly cited here, after rotation components may no longer be hierarchically ordered (1st is the strongest, 2nd is next behind...). But in whole (multivariately), no information is lost in rotation; so the scores of all the extracted components together are as valuable after rotation as before rotation. $\endgroup$ – ttnphns Apr 14 '13 at 17:16
  • $\begingroup$ I just needed to be sure about something and I have posted my workflow in R. $\endgroup$ – Fedja Blagojevic Apr 14 '13 at 17:26
  • $\begingroup$ Yes indeed I use PCA as EFA, but also I have tried with EFA. $\endgroup$ – Fedja Blagojevic Apr 14 '13 at 17:44
7
$\begingroup$

Standardized (to unit variance) principal components after an orthogonal rotation, such as varimax, are simply rotated standardized principal components (by "principal component" I mean PC scores). In linear regression, scaling of individual predictors has no effect and replacing predictors by their linear combinations (e.g. via a rotation) has no effect either. This means that using any of the following in a regression:

  • "raw" principal components (projections on the cov. matrix eigenvectors),
  • standardized principal components,
  • rotated [standardized] principal components,
  • arbitrarily scaled rotated [standardized] principal components,

would lead to exactly the same regression model with identical $R^2$, predictive power, etc. (Individual regression coefficients will of course depend on the normalization and rotation choice.)

The total variance captured by the raw and by the rotated PCs is the same.

This answers your main question. However, you should be careful with your workflows, as it is very easy to get confused and mess up the calculations. The simplest way to obtain standardized rotated PC scores is to use psych::principal function:

 psych::principal(data, rotate="varimax", nfactors=k, scores=TRUE)

Your workflow #2 can be more tricky than you think, because loadings after varimax rotation are not orthogonal, so to obtain the scores you cannot simply project the data onto the rotated loadings. See my answer here for details:

Your workflow #3 is probably also wrong, at least if you refer to the psych::fa function. It does not do PCA; the fm="pa" extraction method refers to "principal factor" method which is based on PCA, but is not identical to PCA (it is an iterative method). As I wrote above, you need psych::principal to perform PCA.

See my answer in the following thread for a detailed account on PCA and varimax:

$\endgroup$
  • $\begingroup$ Thanks for the kind words, @Cbhihe. I am pretty sure that mods can not mark any answer as accepted. If you want to reach the OP (Fedja), you should comment under his question and not under my answer; then he will get notified of your comment. It looks like he is still active on SE ("last seen" on SO today). $\endgroup$ – amoeba Oct 2 '18 at 16:02
  • $\begingroup$ @Mods, can you make this answer "accepted" without OP accruing points (for good measure, since OP has neglected to do so) ? It would be a service to the community. If it is not possible for mods to do this, I will start a thread in meta to discuss this kind of situation where a VERY good answer is neglected by OP and therefore rendered less visible. Last but not least: thanks, amoeba. +1 $\endgroup$ – Cbhihe Oct 2 '18 at 16:16
  • $\begingroup$ @Cbhihe It really is not possible. So if you want to start a Meta thread, you can go ahead, it might be an interesting discussion. $\endgroup$ – amoeba Oct 2 '18 at 16:37
  • $\begingroup$ you're right about mods not ready to act instead of OP in that case. Quite a few Meta-posts exist on that topic and the issue has gone on for as long as SE has been in existence, as far as I can tell. A real problem without a good solution. Pity. Cheers. $\endgroup$ – Cbhihe Oct 2 '18 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.