How does "Fundamental Theorem of Factor Analysis" apply to PCA, or how are PCA loadings defined?

Question

I'm currently going through a slide set I have for "factor analysis" (PCA as far as I can tell).

In it, the "fundamental theorem of factor analysis" is derived which claims that the correlation matrix of the data going into the analysis ($\bf R$) can be recovered using the matrix of factor loadings ($\bf A$):

$$\bf R = AA^\top$$

This however confuses me. In PCA the matrix of "factor loadings" is given by the matrix of eigenvectors of the covariance/correlation matrix of the data (since we're assuming that the data have been standardized, they are the same), with each eigenvector scaled to have length one. This matrix is orthogonal, thus $\bf AA^\top = I$ which is in general not equal to $\bf R$.

Answer 1

This is a reasonable question (+1) that stems from the terminological ambiguity and confusion.

In the context of PCA, people often call principal axes (eigenvectors of the covariance/correlation matrix) "loadings". This is sloppy terminology. What should rather be called "loadings" in PCA, are principal axes scaled by the square roots of the respective eigenvalues. Then the theorem you are referring to will hold.

Indeed, if the eigen-decomposition of the correlation matrix is $$\mathbf R = \mathbf V \mathbf S \mathbf V^\top$$ where $\mathbf V$ are eigenvectors (principal axes) and $\mathbf S$ is a diagonal matrix of eigenvalues, and if we define loadings as $$\mathbf A = \mathbf V \mathbf S^{1/2},$$ then one can easily see that $$\mathbf R = \mathbf A \mathbf A^\top.$$ Moreover, the best rank-$r$ approximation to the correlation matrix is given by the first $r$ PCA loadings: $$\mathbf R \approx \mathbf A_r \mathbf A_r^\top.$$

Please see my answer here for more about reconstructing covariance matrices with factor analysis and PCA loadings.