8
$\begingroup$

Under which conditions can principal components analysis (PCA) and factor analysis (FA) be expected to yield similar results?

$\endgroup$
  • $\begingroup$ Let $\bf L$ be the loadings (not eigenvectors) of the p-m last principal components - those that you drop in PCA (p is the number of variables and m is the number of of components or factors you decided to extract). If $\bf LL'$ is nearly diagonal, then your PCA results are similar to FA results. Some questions for you to read: stats.stackexchange.com/q/123063/3277, stats.stackexchange.com/q/94048/3277. $\endgroup$ – ttnphns Dec 4 '14 at 7:24
  • 1
    $\begingroup$ To put it in other words: when PCA happens to isolate variable-specific noise from signal (common factors) as succesfully as factor analysis regularly does it. PCA, unlike FA, is not intended to do this job, however under some conditions it often appears to do it. Some of these conditions: 1) p is large; 2) noise is small for all variables; 3) noise is about equal for all variables. $\endgroup$ – ttnphns Dec 4 '14 at 8:20
10
$\begingroup$

This is an excellent question, but unfortunately (or maybe fortunately?) I have only recently written a very long answer in a related thread, addressing your question almost exactly. I would kindly ask you to look there and see if that answers your question.

Very briefly, if we just focus on PCA and FA loadings $\mathbf W$, then the difference is that PCA finds $\mathbf W$ to reconstruct the sample covariance (or correlation) matrix $\mathbf C$ as close as possible: $$\mathbf C \approx \mathbf W \mathbf W^\top,$$ whereas FA finds $\mathbf W$ to reconstruct the off-diagonal part of the covariance (or correlation) matrix only: $$\mathrm{offdiag}\{\mathbf C\} \approx \mathbf W \mathbf W^\top.$$ By this I mean that FA does not care what values $\mathbf W \mathbf W^\top$ has on the diagonal, it only cares about the off-diagonal part.

With this in mind, the answer to your question becomes easy to see. If the number $n$ of variables (size of $\mathbf C$) is large, then the off-diagonal part of $\mathbf C$ is almost the whole matrix (diagonal has size $n$ and the whole matrix size $n^2$, so the contribution of the diagonal is only $1/n \to 0$), and so we can expect that PCA approximates FA well. If the diagonal values are rather small, then again they don't play much role for PCA, and PCA ends up being close to FA, exactly as @ttnphns said above.

If, on the other hand, $\mathbf C$ is either small or strongly dominated by the diagonal (in particular if it has very different values on the diagonal), then PCA will have to bias $\mathbf W$ towards reproducing the diagonal as well, and so will end up being quite different from FA. One example is given in this thread:

$\endgroup$
  • $\begingroup$ In your answer, you state that ``minimizing $||C−WW^T−\Psi||^2$'' yields the factor analysis loadings. (I interpret $||\bullet||^2$ as the squared Frobenius norm.) Where can I find a proof for this statement? For PCA, this follows from the Eckart-Young theorem, but I can't see how this applies to FA. $\endgroup$ – stats Dec 5 '14 at 22:37
  • $\begingroup$ Related, ttnphns claims that minimizing $||X−X_k||^2$ is equivalent to minimizing $ ||X^TX−X_k^TX_k||^2$. How can this be shown? $\endgroup$ – stats Dec 5 '14 at 22:43
  • $\begingroup$ To your first question. Yes, it's Frobenius norm. Unlike PCA, FA is more of a framework than one precisely defined method; there are different "methods of factor extraction", resulting in non-identical results. So of course there cannot be any proof for all versions of FA. However, one of the oldest/simplest/widespread methods is to find $W$ and $\Psi$ directly by minimizing this cost function (initialize $\Psi$ randomly, solve for $W$ via PCA, update $\Psi$, etc. until convergence). This is called "iterated principal factor" method, or smth like that. Then nothing remains to be proved :) $\endgroup$ – amoeba Dec 5 '14 at 23:26
  • $\begingroup$ To your second question. Not sure if this is true in general (maybe it is, maybe not), but I never use it in my linked answer. Look at my "Update 2" carefully, this statement is not needed. $\endgroup$ – amoeba Dec 5 '14 at 23:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.