When should we use PCA over factor analysis? Aren't they essentially the same thing except that factor analysis is modeling observed variables as linear combinations of unobserved factors? Whereas PCA is modeling components as linear combinations of observed variables?

  • 2
    $\begingroup$ This is a slightly contentious field, but one widespread interpretation of PCA is that it is not a model-based technique at all. It is a just a transformation technique. There is no error term and no estimation. It's true that many factor analysis enthusiasts regard PCA as a limiting or even degenerate case of factor analysis and there are family resemblances but "essentially the same thing" is likely to be too strong a statement to get much support. $\endgroup$ – Nick Cox Dec 17 '13 at 16:44
  • 1
    $\begingroup$ 1) They both model observed variables as linear combinations of latent variables. 2) In PCA, the opposite (a component is a linear combination of the variables) is also true, but not in FA. 3) Point (1) implies error term, like regression. With FA, the error terms for different variables are constrained to be orthogonal; they are called "unique factors". 4) FA aims to explain, with few common factors, most of pairwise covariances (correlations), PCA aims to explain, with few pr. components, most of multivariate variability. 5) Despite these big differences, PCA may be considered a case of FA. $\endgroup$ – ttnphns Dec 17 '13 at 17:05
  • 2
    $\begingroup$ Helpful, but underlines my point that this field is slightly contentious. For example, I suggest that 4) confuses two distinct issues. PCA has no aim except to re-express all the variability in terms of orthogonal components. It's users of PCA who often (but not always) also hope that a few components work to "explain" most of the variability. $\endgroup$ – Nick Cox Dec 17 '13 at 17:10
  • $\begingroup$ @Nick, yes it is contentious field. As for point (4), I think it is fine. If PCA and FA can ever be compared (and the OP demands comparison) PCA must reflect the human aim because FA is all about it. PCA without that aim to "explain" looks no better than an arbitrary orthogonal rotation. $\endgroup$ – ttnphns Dec 17 '13 at 18:46
  • $\begingroup$ I'd turn that round completely. One of the attractions of PCA -- to people attracted to it -- is the lack of arbitrary decisions in model-building. Naturally, one must choose data to feed to PCA, in all senses. $\endgroup$ – Nick Cox Dec 17 '13 at 19:16

Browse other questions tagged or ask your own question.