61

This question is largely about definitions of PCA/FA, so opinions might differ. My opinion is that PCA+varimax should not be called either PCA or FA, bur rather explicitly referred to e.g. as "varimax-rotated PCA". I should add that this is quite a confusing topic. In this answer I want to explain what a rotation actually is; this will require some ...


43

Reason for rotation. Rotations are done for the sake of interpretation of the extracted factors in factor analysis (or components in PCA, if you venture to use PCA as a factor analytic technique). You are right when you describe your understanding. Rotation is done in the pursuit of some structure of the loading matrix, which may be called simple structure. ...


25

"Rotations" is an approach developed in factor analysis; there rotations (such as e.g. varimax) are applied to loadings, not to eigenvectors of the covariance matrix. Loadings are eigenvectors scaled by the square roots of the respective eigenvalues. After the varimax rotation, the loading vectors are not orthogonal anymore (even though the rotation is ...


20

This answer succeeds this general question on rotations in factor analysis (please read it) and briefly describes a number of specific methods. Rotations are performed iteratively and on every pair of factors (columns of the loading matrix). This is needed because the task to optimize (maximize or minimize) the objective criterion simultaneously for all the ...


16

This is going to be a non-technical answer. You are right: PCA is essentially a rotation of the coordinate axes, chosen such that each successful axis captures as much variance as possible. In some disciplines (such as e.g. psychology), people like to apply PCA in order to interpret the resulting axes. I.e. they want to be able to say that principal axis #...


14

This answer is to present, in a path chart form, things about which @amoeba reasoned in his deep (but slightly complicated) answer on this thread (I'm a kind of agree with it by 95%) and how they appear to me. PCA in its proper, minimal form is the specific orthogonal rotation of correlated data to its uncorrelated form, with the principal components ...


12

Who told you that factor loadings can't be greater than 1? It can happen. Especially with highly correlated factors. This passage from a report about it by a prominent pioneer of SEM pretty much sums it up: "This misunderstanding probably stems from classical exploratory factor analysis where factor loadings are correlations if a correlation matrix is ...


11

It is not completely clear to me that what you are asking is what you really need: a common preprocessing step in machine learning is dimensionality reduction + whitening, which means doing PCA and standardizing the components, nothing else. But I will nevertheless focus on your question as it is formulated, because it's more interesting. Let $\mathbf X$ ...


9

Loading in factor analysis or in PCA (see 1, see 2, see 3) is the regression coefficient, weight in a linear combination predicting variables (items) by standardized (unit-variance) factors/components. Reasons for a loading to exceed $1$: Reason 1: analyzed covariance matrix. If analyzed were standardized variables, that is, the analysis was based on ...


8

Yes, there may be a reason to withdraw from rotation in factor analysis. That reason is actually similar to why we usually do not rotate principal components in PCA (i.e. when we use it primarily for dimensionality reduction and not to model latent traits). After extraction, factors (or components) are orthogonal$^1$ and are usually output in descending ...


7

You are correct. Stata is weird about this. Stata gives different results from SAS, R and SPSS, and it is difficult (in my opinion) to understand why without delving quite deep into the world of factor analysis and PCA. Here's how you know that something weird is happening. The sum of the squared loadings for a component are equal to the eigenvalue for ...


7

Standardized (to unit variance) principal components after an orthogonal rotation, such as varimax, are simply rotated standardized principal components (by "principal component" I mean PC scores). In linear regression, scaling of individual predictors has no effect and replacing predictors by their linear combinations (e.g. via a rotation) has no effect ...


6

As the author of the psych package I will try to explain what is going on and offer a solution. First, as @ttnhpns correctly points out, rotated principal components are no longer principal components, they are merely components. @ttnhpns is also correct that we should not call the sums of squares of these rotated components "eigen values" but rather "...


6

There are several (at least two) potentially confusing issues cropping up here. Confusing issue #1: psych::principal() uses varimax rotation by default The principal function is part of the psych package that is mostly focusing on doing factor analysis. Even though the principal function is doing PCA, it is following the FA-style approach. In particular, ...


6

The two citations do not generally contradict each other and both look to me correct. The only underwork is in Perhaps you mean sum of squared loadings for a principal component, after rotation one should better drop word "principal" since rotated components or factors are not "principal" anymore, to be rigorous. Also (important!) the second citation is ...


6

Both PCA and PAF can be seen as ways of dimension reduction. In discussing their differences, I'll be relying on Exploratory Factor Analysis by Fabrigar and Wegener (2012). I'm not going to get too deep into the math or computational algorithms for this stuff; I'll keep it at a high level. Principal component analysis (PCA) The goal is to create variables (...


5

I think that the answer to your question is Yes (at least, in the big picture sense). Should you be wanting to dive deeper into details, I would suggest you to review this excellent discussion here on Cross Validated, especially an answer by @amoeba and/or Chapter 6 of the excellent online book by Revelle (2015). Having said that, I would like to make the ...


5

I rerun your analysis in SPSS (I don't have Stata, and I didn't rerun it in Matlab this time). The sweet pulp of your mistaken analysis is that you somehow managed to rotate eigenvectors, whereas rotations are normaly done of loadings. Please read my recent answers about eigenvectors/loadings and about rotations. Your first analysis extracted all 5 ...


5

Yes. That's exactly what principal component regression is: https://en.wikipedia.org/wiki/Principal_component_regression. No need to rotate. In fact, rotating would not make any difference as far as prediction is concerned; see Using varimax-rotated PCA components as predictors in linear regression. Of course if you want to interpret individual regression ...


4

It doesn't make any difference where your model comes from. Lavaan doesn't know that the model comes form an EFA, or that you used oblimin (or any other) rotation. You should always include correlations between your factors, unless you have a very good reason to believe that they are correlated zero. Lavaan includes factor covariances (and factor ...


4

Do we need factor rotation? Of all the factors? Do the strongest unrotated factor reveal the "general factor"? Books do not urge, "rotate, don't leave your factors unrotated". Rather, they say that rotation can benefit in interpretation. Factor rotations are done for the sake of more easy and "better" interpretation of the ...


3

The differences between the Stata PCA methods and the conventional methods used in R or SPSS are: 1. Scaling eigenvectors/components Stata rotates eigenvectors. Whereas, R or SPSS PCA-rotation methods normally rotates after scaling eigenvectors by the sqrt of the eigenvalues to produce the component loadings more typical in factor analysis. 2. Convergence ...


3

Orthogonal rotations are special cases of oblique rotations, so yes, they can show up. (Can you provide better links to your articles?) Edit: I don't think that the Bandalos and Boehn-Haufman says what you say it said. E.g. the end of that section of the chapter says [if you have done both orthogonal and oblique rotations] "the results from the oblique ...


3

An overview of methods to compute component and factor scores notices on the so called "standardized" factor scores, The scores computed ... are scaled: they have variances equal to or close to 1 (standardized or near standardized) - not the true factor variances (which equal the sum of squared structure loadings). and In FA (not PCA), ...


2

Although this question has already an accepted answer I'd like to add something to the point of the question. "PCA" -if I recall correctly - means "principal components analysis"; so as long as you're analyzing the principal components, may it be without rotation or with rotation, we are still in the analysis of the "principal components" (...


2

Factor analysis rotates loadings, not eigenvectors; see my answer here for a lengthy discussion: Is PCA followed by a rotation (such as varimax) still PCA? Your V are eigenvectors, and loadings are given by V*sqrt(D), so what you need to do is rotatefactors(V*sqrt(D)). But it's better to make sure that the zero column of V is kicked out, otherwise Matlab ...


2

It seems (/METHOD=CORRELATION) that you are asking SPSS to perform FA on the correlation matrix (actually by typing /EXTRACTION PC you seem to be performing PCA and not FA, but it does not matter for this question). Correlation matrix does not change if you scale individual variables, so any amount of normalizing or standardizing would not change it. You ...


2

In his perspicacious and comprehensive answer @amoeba has shown - as part of the answer - how one can rotate two uncorrelated variables (such as principal components for example) to achieve the wanted variances for them (while at expense of losing uncorrelatedness, of course). Let orthogonal variables $X$ and $Y$ have variances $\sigma^2_{max}$ (a larger) ...


2

If appears that fa defaults to iterated principal factors. So, to be somewhat careful in this: If you want a principal factors solution with priors based on Squared multiple correlations (and not iterated), you code in SAS would be: proc factor n=4 method=prin rotate=none; priors smc; var your-variables-here; run; and the equivalent code in R would be: ...


2

As Jeremy mentioned in the comments, if your goal is prediction, then there is no sense in creating an interpretable factor. You can use flexible regression or machine learning methods directly on the items. Adding a factor structure only limits the relationship between the items and the outcome by imposing independence constraints, which seem unnecessary to ...


Only top voted, non community-wiki answers of a minimum length are eligible