Is it valid to use only some of the components as regressors from a PCA?

Question

I was trying a 4 components PCA where the 1st and 2nd components have emerged exactly as I expected, but the other two components do not commensurate with the theory. I mean, I expected a little different kind of loads. Some of the variables expected to belong to component 3 have actually loaded highly on component 4 and some expected to belong to component 4 have loaded on component 3. I have tried in different ways, but things don't really improve and theoretically component 3 and component 4 do not make much sense to me. You can have a view of the loadings table here-

Now, I am planning to use the 1st and 2nd components as regressors (as they have clearly emerged) along with the 9 individual variables that have loaded highly on the other two components (because their linear combinations don't hold to be meaningful) in a further regression on a dependent variable. My questions are-

1) Is it statistically valid to use only component 1 and 2 (component scores, basically) along with the 9 individual variables of component 3 and 4 as regressors in a regression?

2) Is it possible to include a few other variables as regressors too which were not included in the PCA?

Answer 1

This is a complicated question. First of all, when you are interested in interpreting your components in meaningful ways, you should be aware that principle components analysis is only useful for telling you how many factors to extract from your data. Generally, after a PCA, you plot the variances of the components in order from largest to smallest.

There are a wide variety of ways of making the decision as to how many factors to extract (see, for example, Zwick & Velicer, 1986). One of the easiest and most widely used is the "scree test", a graphical approach in which you plot the size of the variances of each of your components in descending order. You then identify the "elbow" of the plot, or the place in the plot where the size of the components tapers off dramatically, and where the remaining components account for approximately equal amounts of variance. So, in the plot below, we would identify the third component as the elbow and choose to extract two components.

Once you have made a decision about how many components to extract, you should choose how to rotate your components (so that you know which items load on each extracted factor). In general, rotations that allow your extracted factors to correlate (such as the oblimin rotation) result in more interpretable solutions than rotations that do not (such as the varimax). After all, how often does it occur that two extracted factors are exactly orthogonal (which is enforced in varimax or other orthogonal rotations).

Overall, though, I would bear in mind that the process of Exploratory Factor Analysis (i.e., deciding how many factors to extract through PCA, choosing a rotation, interpreting your resultant factors and factor loadings) is an iterative process (may I say exploratory) process; if a given solution is uninterpretable, it can be perfectly valid to choose to extract a different number of factors to see if that yields a solution that is more interpretable. Exploratory Factor Analysis is NOT Confirmatory Factor Analysis, nor should it be treated as such; you are not testing the adequacy of a given measurement model using some test of significance. Instead, you are attempting to pull structure out of a set of data in such a way that the structure has a valid theoretical interpretation. In the end, the final criterion by which you should judge your EFA is interpretability, rather than some metric of model fit.