What is the difference between principal component analyses (PCA) and principal axis factoring (PAF)?

Also, I understand the difference between varimax and oblimin rotations, but is that the same as orthogonal and oblique?

  • $\begingroup$ It is pity that you didn't make a search before asking your question. Difference and similarity between PAF and PCA extractions are clear from this description. Step-by-step computational comparisons between the two are demonstrated here. Differences between PCA and FA models in general were discussed in many good places on the site, I'm linking just to my answer here. $\endgroup$ – ttnphns Jun 11 '17 at 18:28
  • $\begingroup$ For rotations, see e.g. stats.stackexchange.com/q/185216/3277, stats.stackexchange.com/q/151653/3277 $\endgroup$ – ttnphns Jun 11 '17 at 18:29

Both PCA and PAF can be seen as ways of dimension reduction. In discussing their differences, I'll be relying on Exploratory Factor Analysis by Fabrigar and Wegener (2012). I'm not going to get too deep into the math or computational algorithms for this stuff; I'll keep it at a high level.

Principal component analysis (PCA)

  • The goal is to create variables (components) that maximize interindividual variance—that is, try to create an index where people differ most.
  • Components are always orthogonal—each component explains non-redundant information.
  • Components are linear combination of indicators: items ($x$) predict components ($z$): $x \rightarrow z$
  • We are trying to explain variances in variables; components account for maximal variance in observed variables.

Principal axis factoring (PAF)

  • This is an exploratory factor analysis (EFA) approach. Here, we want a parsimonious representation of observed correlations between variables by latent factors. I'll talk about stuff general to EFA real quick, because it helps differentiate PAF from PCA.
  • In EFA, we are operating under the theory of the common factor model. In this model, we assume that there are latent constructs out there—things that exist but we cannot measure directly (especially psychological constructs, like anxiety or prejudice). We assume that variables ($x$) are linear combinations of latent factors ($\Psi$)and residuals ($\epsilon$): $\epsilon \rightarrow x \leftarrow \Psi$.
  • In EFA, factor scores at not real scores, instead they are estimates of the underlying latent constructs.

So what is unique about PAF among other types of EFA?

  • Instead of using the correlation matrix among variables, we use a reduced correlation matrix. In a regular correlation matrix, the diagonals are all "1". In a reduced correlation matrix, we replace the 1s with the communalities of observed variables. Communalities are how much variance in the item is explained by the factor structure.
  • So now you may be wondering: Wait, we get communalities after we run the factor analysis. How can we run a PAF EFA where communalities are on the diagonals when we don't have the communalities yet? PAF uses an iterative procedure: We start with replacing the 1s with the squared multiple correlation (SMC) of the variable (that is, how much of the variance in that variable that can be explained by the rest of the variables in the correlation matrix). Then we do the EFA: Calculate eigenvalues and loadings, estimate communalities. Then we replace the SMC with the new communalities and do the EFA again, getting new eigenvalues, loadings and communalities. We keep doing this until the communalities do not change much; then the loadings are final.

So when to use PCA or PAF?

  • Other people might have different opinions, but this is how I have been trained to think (as a social psychologist who focuses on quantitative methods): We use a PCA when the theory behind the index variable is that the index is an outcome of the indicators. Think about socioeconomic status. We do not think there is some latent thing called "socioeconomic status" out there that we cannot directly measure. Instead, we might say socioeconomic status ($z$) is a linear combination of education ($x_1$) and income ($x_2$): $z = x_1 + x_2$.

  • Now consider prejudice. We cannot directly measure this. Instead, we give people a bunch of questions to answer, and we assume that there is some latent construct called "prejudice" that is influencing how people answer these questions. This is a perfect time to use principal axis factoring (or another way of getting at an EFA, such as maximum likelihood). So we might ask people how much they agree with the items "I feel negatively toward Black people" ($x_1$) and "I cannot imagine falling in love with a Black person" ($x_2$). We assume that prejudice ($\Psi$) and residuals ($\epsilon$) predict these two items:

    $$x_1 = \Psi + \epsilon_1$$


    $$x_2 = \Psi + \epsilon_2$$

Varimax vs. oblimin

  • As you mentioned, we can use an orthogonal or oblique rotation when we do principal axis factoring (or other exploratory factor analyses).
  • Varimax is one of the methods for orthogonal rotation, and probably the most popular and valid.
  • Oblimin is one of the methods for oblique rotations, and there are multiple ways of doing oblimin rotations! Direct oblimin is usually what people use.
  • So, varimax and oblimin are not the same thing as orthogonal and oblique, respectively; instead, they are both types of orthogonal and oblique rotations.
  • 1
    $\begingroup$ +1. A very neat and simple answer. Just I want to scratch off a tiny nuance about that a little bit dogmatic/narrow use a PCA when the theory behind the index variable is that the index is an outcome of the indicators, we assume that there is some latent construct called "prejudice" that is influencing how people answer these questions. This is a perfect time for EFA. This stance would be perfect if we new the substance ("prejudice") of the latent factor in advance. $\endgroup$ – ttnphns Jun 11 '17 at 19:54
  • 1
    $\begingroup$ (cont.) We actually don't, in EFA (while we do in a questionnaire/test creation process in general). Interpretation "prejudice" is what might follow only after factor(s) is extracted (vs regression, where we know both the predictors and the predictands). Before, we only know "there must be some factor behind". Therefore, factor is a (initially abstract) construct like a component: it is extracted from the variables, not preconceived. $\endgroup$ – ttnphns Jun 11 '17 at 19:54
  • 1
    $\begingroup$ (cont.) Well, it is preconceived as a factor, predictor (and is modeled accordingly statistically not to be a function of variables), still, since its contents/meaning (i.e. loadings) isn't known a priori, it is a function of variables: specifically, their correlations. That reasoning somewhat dissolves borderline between a component and a factor: both are just construct. One is more descriptive/aggregative, the other is more theoretic/causal. $\endgroup$ – ttnphns Jun 11 '17 at 19:55
  • 1
    $\begingroup$ (cont.) PCA often gives components which are roughly/tentatively treated as latent factors, and as variables increase it approaches to FA in results. Also, both in FA and PCA variables are linear functions (+error) of the common constructs. Finally, factor scores are computed as linear functions of variables. $\endgroup$ – ttnphns Jun 11 '17 at 19:55
  • 1
    $\begingroup$ (cont.) All those are the bases why PCA is often called the "simplest method of EFA". Despite their well known model difference. Common EFA is a specially constrained development of PCA, and indeed, there are in-between procedures (such as probabilistic PCA, image factor analysis...). These were just bla-bla notes on your otherwise perfect contribution. $\endgroup$ – ttnphns Jun 11 '17 at 19:56

I think that there is one large difference between PCA and PFA that is tacitly mentioned in any discussion and is not usually explicated. I make this an answer rather than a comment, because it was a source of confusion to me for a long time. I also hope that by answering, others will chime in.
PCA is an axiomatic procedure. It will in practice produce new variables that absolutely determined. That is up to choice of a sign and choice of an ordering it is unique. In PCA, one is choosing variables by how much of the data they explain. PFA and it's like minded ilk are different in that one is looking not to describe variance, but rather some underlying, but unknown effect. The example Mark White gives is excellent. To continue, let's assume that we have variables 'education level', 'integration pct in ones hometown', and 'income'. Further they are used to predict what percentage Let' pretened that they are orthogonal (independent). Then prejudice would be some unknown linear combination of these variables. It's conceivable, at least in an ideal society, that education level is a better predictor of who one might want as a tenant. If so, creating the prejudice variable might lead to a model that is less predictive. In addition, suppose one finds a combination of variables that one thinks is prejudice. Short of by fiat defining prejudice as the variable chosen, one can't be quite sure what one has defined. All one can say is that one has found a factor that looks like prejudice. Reading a stat book about confounding variables (Friedman, Pisana et.al. comes to mind), will make one realize how complicated such an endeavor as PFA truly can be.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.