I'm currently going through a slide set I have for "factor analysis" (PCA as far as I can tell).

In it, the "fundamental theorem of factor analysis" is derived which claims that the correlation matrix of the data going into the analysis ($\bf R$) can be recovered using the matrix of factor loadings ($\bf A$):

$$\bf R = AA^\top$$

This however confuses me. In PCA the matrix of "factor loadings" is given by the matrix of eigenvectors of the covariance/correlation matrix of the data (since we're assuming that the data have been standardized, they are the same), with each eigenvector scaled to have length one. This matrix is orthogonal, thus $\bf AA^\top = I$ which is in general not equal to $\bf R$.

  • $\begingroup$ In addition to @amoeba's answer, look in my answer addresing the terminologic ambiguety. I do not recommend to call eigenvectors matrix A (which are loadings), for reasons of clarity. The (right-side) eigenvector matrix is usually labeled V (because R=USV' by svd), not A. Another equivalent name (coming from biplot terminology) for eigenvectors is "standard coordinates", and for loadings is "principal coordinates". $\endgroup$ – ttnphns Nov 27 '14 at 18:26
  • $\begingroup$ ("standard coordinates" - because the inertia, or scale of the eigenvalues, is unit magnitude when endowing them; "principal coordinates" - because it is original full magnitude when endowing them.) $\endgroup$ – ttnphns Nov 27 '14 at 18:31

This is a reasonable question (+1) that stems from the terminological ambiguity and confusion.

In the context of PCA, people often call principal axes (eigenvectors of the covariance/correlation matrix) "loadings". This is sloppy terminology. What should rather be called "loadings" in PCA, are principal axes scaled by the square roots of the respective eigenvalues. Then the theorem you are referring to will hold.

Indeed, if the eigen-decomposition of the correlation matrix is $$\mathbf R = \mathbf V \mathbf S \mathbf V^\top$$ where $\mathbf V$ are eigenvectors (principal axes) and $\mathbf S$ is a diagonal matrix of eigenvalues, and if we define loadings as $$\mathbf A = \mathbf V \mathbf S^{1/2},$$ then one can easily see that $$\mathbf R = \mathbf A \mathbf A^\top.$$ Moreover, the best rank-$r$ approximation to the correlation matrix is given by the first $r$ PCA loadings: $$\mathbf R \approx \mathbf A_r \mathbf A_r^\top.$$

Please see my answer here for more about reconstructing covariance matrices with factor analysis and PCA loadings.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.