Under which conditions can principal components analysis (PCA) and factor analysis (FA) be expected to yield similar results?
$\begingroup$
$\endgroup$
2
1 Answer
$\begingroup$
$\endgroup$
4
This is an excellent question, but unfortunately (or maybe fortunately?) I have only recently written a very long answer in a related thread, addressing your question almost exactly. I would kindly ask you to look there and see if that answers your question.
Very briefly, if we just focus on PCA and FA loadings $\mathbf W$, then the difference is that PCA finds $\mathbf W$ to reconstruct the sample covariance (or correlation) matrix $\mathbf C$ as close as possible: $$\mathbf C \approx \mathbf W \mathbf W^\top,$$ whereas FA finds $\mathbf W$ to reconstruct the off-diagonal part of the covariance (or correlation) matrix only: $$\mathrm{offdiag}\{\mathbf C\} \approx \mathbf W \mathbf W^\top.$$ By this I mean that FA does not care what values $\mathbf W \mathbf W^\top$ has on the diagonal, it only cares about the off-diagonal part.
With this in mind, the answer to your question becomes easy to see. If the number $n$ of variables (size of $\mathbf C$) is large, then the off-diagonal part of $\mathbf C$ is almost the whole matrix (diagonal has size $n$ and the whole matrix size $n^2$, so the contribution of the diagonal is only $1/n \to 0$), and so we can expect that PCA approximates FA well. If the diagonal values are rather small, then again they don't play much role for PCA, and PCA ends up being close to FA, exactly as @ttnphns said above.
If, on the other hand, $\mathbf C$ is either small or strongly dominated by the diagonal (in particular if it has very different values on the diagonal), then PCA will have to bias $\mathbf W$ towards reproducing the diagonal as well, and so will end up being quite different from FA. One example is given in this thread:
-
$\begingroup$ In your answer, you state that ``minimizing $||C−WW^T−\Psi||^2$'' yields the factor analysis loadings. (I interpret $||\bullet||^2$ as the squared Frobenius norm.) Where can I find a proof for this statement? For PCA, this follows from the Eckart-Young theorem, but I can't see how this applies to FA. $\endgroup$– statsCommented Dec 5, 2014 at 22:37
-
$\begingroup$ Related, ttnphns claims that minimizing $||X−X_k||^2$ is equivalent to minimizing $ ||X^TX−X_k^TX_k||^2$. How can this be shown? $\endgroup$– statsCommented Dec 5, 2014 at 22:43
-
$\begingroup$ To your first question. Yes, it's Frobenius norm. Unlike PCA, FA is more of a framework than one precisely defined method; there are different "methods of factor extraction", resulting in non-identical results. So of course there cannot be any proof for all versions of FA. However, one of the oldest/simplest/widespread methods is to find $W$ and $\Psi$ directly by minimizing this cost function (initialize $\Psi$ randomly, solve for $W$ via PCA, update $\Psi$, etc. until convergence). This is called "iterated principal factor" method, or smth like that. Then nothing remains to be proved :) $\endgroup$– amoebaCommented Dec 5, 2014 at 23:26
-
$\begingroup$ To your second question. Not sure if this is true in general (maybe it is, maybe not), but I never use it in my linked answer. Look at my "Update 2" carefully, this statement is not needed. $\endgroup$– amoebaCommented Dec 5, 2014 at 23:28
p-mlast principal components - those that you drop in PCA (pis the number of variables andmis the number of of components or factors you decided to extract). If $\bf LL'$ is nearly diagonal, then your PCA results are similar to FA results. Some questions for you to read: stats.stackexchange.com/q/123063/3277, stats.stackexchange.com/q/94048/3277. $\endgroup$pis large; 2) noise is small for all variables; 3) noise is about equal for all variables. $\endgroup$