What is the difference between PCA and PAF method in factor analysis?

Question

What is the difference between principal component analyses (PCA) and principal axis factoring (PAF)?

Also, I understand the difference between varimax and oblimin rotations, but is that the same as orthogonal and oblique?

Answer 1

Both PCA and PAF can be seen as ways of dimension reduction. In discussing their differences, I'll be relying on Exploratory Factor Analysis by Fabrigar and Wegener (2012). I'm not going to get too deep into the math or computational algorithms for this stuff; I'll keep it at a high level.

Principal component analysis (PCA)

• The goal is to create variables (components) that maximize interindividual variance—that is, try to create an index where people differ most.
• Components are always orthogonal—each component explains non-redundant information.
• Components are linear combination of indicators: items ($x$) predict components ($z$): $x \rightarrow z$
• We are trying to explain variances in variables; components account for maximal variance in observed variables.

Principal axis factoring (PAF)

• This is an exploratory factor analysis (EFA) approach. Here, we want a parsimonious representation of observed correlations between variables by latent factors. I'll talk about stuff general to EFA real quick, because it helps differentiate PAF from PCA.
• In EFA, we are operating under the theory of the common factor model. In this model, we assume that there are latent constructs out there—things that exist but we cannot measure directly (especially psychological constructs, like anxiety or prejudice). We assume that variables ($x$) are linear combinations of latent factors ($\Psi$)and residuals ($\epsilon$): $\epsilon \rightarrow x \leftarrow \Psi$.
• In EFA, factor scores at not real scores, instead they are estimates of the underlying latent constructs.

So what is unique about PAF among other types of EFA?

• Instead of using the correlation matrix among variables, we use a reduced correlation matrix. In a regular correlation matrix, the diagonals are all "1". In a reduced correlation matrix, we replace the 1s with the communalities of observed variables. Communalities are how much variance in the item is explained by the factor structure.
• So now you may be wondering: Wait, we get communalities after we run the factor analysis. How can we run a PAF EFA where communalities are on the diagonals when we don't have the communalities yet? PAF uses an iterative procedure: We start with replacing the 1s with the squared multiple correlation (SMC) of the variable (that is, how much of the variance in that variable that can be explained by the rest of the variables in the correlation matrix). Then we do the EFA: Calculate eigenvalues and loadings, estimate communalities. Then we replace the SMC with the new communalities and do the EFA again, getting new eigenvalues, loadings and communalities. We keep doing this until the communalities do not change much; then the loadings are final.

So when to use PCA or PAF?

• Other people might have different opinions, but this is how I have been trained to think (as a social psychologist who focuses on quantitative methods): We use a PCA when the theory behind the index variable is that the index is an outcome of the indicators. Think about socioeconomic status. We do not think there is some latent thing called "socioeconomic status" out there that we cannot directly measure. Instead, we might say socioeconomic status ($z$) is a linear combination of education ($x_1$) and income ($x_2$): $z = x_1 + x_2$.

• Now consider prejudice. We cannot directly measure this. Instead, we give people a bunch of questions to answer, and we assume that there is some latent construct called "prejudice" that is influencing how people answer these questions. This is a perfect time to use principal axis factoring (or another way of getting at an EFA, such as maximum likelihood). So we might ask people how much they agree with the items "I feel negatively toward Black people" ($x_1$) and "I cannot imagine falling in love with a Black person" ($x_2$). We assume that prejudice ($\Psi$) and residuals ($\epsilon$) predict these two items:

  $$x_1 = \Psi + \epsilon_1$$

  and

  $$x_2 = \Psi + \epsilon_2$$

Varimax vs. oblimin

• As you mentioned, we can use an orthogonal or oblique rotation when we do principal axis factoring (or other exploratory factor analyses).
• Varimax is one of the methods for orthogonal rotation, and probably the most popular and valid.
• Oblimin is one of the methods for oblique rotations, and there are multiple ways of doing oblimin rotations! Direct oblimin is usually what people use.
• So, varimax and oblimin are not the same thing as orthogonal and oblique, respectively; instead, they are both types of orthogonal and oblique rotations.

Answer 2

I think that there is one large difference between PCA and PFA that is tacitly mentioned in any discussion and is not usually explicated. I make this an answer rather than a comment, because it was a source of confusion to me for a long time. I also hope that by answering, others will chime in.
PCA is an axiomatic procedure. It will in practice produce new variables that absolutely determined. That is up to choice of a sign and choice of an ordering it is unique. In PCA, one is choosing variables by how much of the data they explain. PFA and it's like minded ilk are different in that one is looking not to describe variance, but rather some underlying, but unknown effect. The example Mark White gives is excellent. To continue, let's assume that we have variables 'education level', 'integration pct in ones hometown', and 'income'. Further they are used to predict what percentage Let' pretened that they are orthogonal (independent). Then prejudice would be some unknown linear combination of these variables. It's conceivable, at least in an ideal society, that education level is a better predictor of who one might want as a tenant. If so, creating the prejudice variable might lead to a model that is less predictive. In addition, suppose one finds a combination of variables that one thinks is prejudice. Short of by fiat defining prejudice as the variable chosen, one can't be quite sure what one has defined. All one can say is that one has found a factor that looks like prejudice. Reading a stat book about confounding variables (Friedman, Pisana et.al. comes to mind), will make one realize how complicated such an endeavor as PFA truly can be.