What is the difference between principal component analyses (PCA) and principal axis factoring (PAF)?
Also, I understand the difference between varimax and oblimin rotations, but is that the same as orthogonal and oblique?
Both PCA and PAF can be seen as ways of dimension reduction. In discussing their differences, I'll be relying on Exploratory Factor Analysis by Fabrigar and Wegener (2012). I'm not going to get too deep into the math or computational algorithms for this stuff; I'll keep it at a high level.
So what is unique about PAF among other types of EFA?
Other people might have different opinions, but this is how I have been trained to think (as a social psychologist who focuses on quantitative methods): We use a PCA when the theory behind the index variable is that the index is an outcome of the indicators. Think about socioeconomic status. We do not think there is some latent thing called "socioeconomic status" out there that we cannot directly measure. Instead, we might say socioeconomic status ($z$) is a linear combination of education ($x_1$) and income ($x_2$): $z = x_1 + x_2$.
Now consider prejudice. We cannot directly measure this. Instead, we give people a bunch of questions to answer, and we assume that there is some latent construct called "prejudice" that is influencing how people answer these questions. This is a perfect time to use principal axis factoring (or another way of getting at an EFA, such as maximum likelihood). So we might ask people how much they agree with the items "I feel negatively toward Black people" ($x_1$) and "I cannot imagine falling in love with a Black person" ($x_2$). We assume that prejudice ($\Psi$) and residuals ($\epsilon$) predict these two items:
$$x_1 = \Psi + \epsilon_1$$
$$x_2 = \Psi + \epsilon_2$$
I think that there is one large difference between PCA and PFA that is tacitly mentioned in any discussion and is not usually explicated. I make this an answer rather than a comment, because it was a source of confusion to me for a long time. I also hope that by answering, others will chime in.
PCA is an axiomatic procedure. It will in practice produce new variables that absolutely determined. That is up to choice of a sign and choice of an ordering it is unique. In PCA, one is choosing variables by how much of the data they explain. PFA and it's like minded ilk are different in that one is looking not to describe variance, but rather some underlying, but unknown effect. The example Mark White gives is excellent. To continue, let's assume that we have variables 'education level', 'integration pct in ones hometown', and 'income'. Further they are used to predict what percentage Let' pretened that they are orthogonal (independent). Then prejudice would be some unknown linear combination of these variables. It's conceivable, at least in an ideal society, that education level is a better predictor of who one might want as a tenant. If so, creating the prejudice variable might lead to a model that is less predictive. In addition, suppose one finds a combination of variables that one thinks is prejudice. Short of by fiat defining prejudice as the variable chosen, one can't be quite sure what one has defined. All one can say is that one has found a factor that looks like prejudice. Reading a stat book about confounding variables (Friedman, Pisana et.al. comes to mind), will make one realize how complicated such an endeavor as PFA truly can be.