PCA scores in multiple regression

Question

A while ago a user on R-help mailing list asked about the soundness of using PCA scores in a regression. The user is trying to use some PC scores to explain variation in another PC (see full discussion here). The answer was that no, this is not sound because PCs are orthogonal to each other. Can someone explain in a bit more detail why this is so?

Answer 1

A principal component is a weighted linear combination of all your factors (X's).

example: PC1 = 0.1X1 + 0.3X2

There will be one component for each factor (though in general a small number are selected).

The components are created such that they have zero correlation (are orthogonal), by design.

Therefore, component PC1 should not explain any variation in component PC2.

You may want to do regression on your Y variable and the PCA representation of your X's, as they will not have multi-collinearity. However, this could be hard to interpret.

If you have more X's than observations, which breaks OLS, you can regress on your components, and simply select a smaller number of the highest variation components.

Principal Component Analysis by Jollife a very in-depth and highly cited book on the subject

This is also good: http://www.statsoft.com/textbook/principal-components-factor-analysis/

Answer 2

Principal components are orthogonal by definition, so any pair of PCs will have zero correlation.

However, PCA can be used in regression if there are a large number of explanatory variables. These can be reduced to a small number of principal components and used as predictors in a regression.

Answer 3

Careful... just because the PCs are by construction orthogonal to each other does not mean that there is not a pattern or that one PC can not appear to "explain" something about the other PCs.

Consider 3D data (X,Y,Z) describing a large number of points distributed evenly on the surface of an American football (it is an ellipsoid -- not a sphere -- for those who have never watched American football). Imagine that the football is in an arbitrary configuration so that neither X nor Y nor Z is along the long axis of the football.

Principal components will place PC1 along the long axis of the football, the axis that describes the most variance in the data.

For any point in the PC1 dimension along the long axis of the football, the planar slice represented by PC2 and PC3 should describe a circle and the radius of this circular slice depends on the PC1 dimension. It is true that regressions of PC2 or PC3 on PC1 should give a zero coefficient globally, but not over smaller sections of the football.... and it is clear that a 2D graph of PC1 and PC2 would show an "interesting" limiting boundary that is two-valued, nonlinear, and symmetric.

Answer 4

This is a bit off-topic, but here's some good stuff about PCA (mostly in R):

HTML

http://www.statsoft.com/textbook/principal-components-factor-analysis/?button=1
http://ordination.okstate.edu/PCA.htm
http://astrostatistics.psu.edu/datasets/R/MV.html
http://www.statmethods.net/advstats/factor.html

---

PDF

http://cran.r-project.org/web/packages/HSAUR/vignettes/Ch_principal_components_analysis.pdf
http://www.uga.edu/strata/software/pdf/pcaTutorial.pdf
http://www.cs.otago.ac.nz/cosc453/student_tutorials/principal_components.pdf
http://www.miislita.com/information-retrieval-tutorial/pca-spca-tutorial.pdf
http://www.unc.edu/~rcm/book/factor.pdf

Answer 5

If your data is high dimensional and noisy, and you don't have a large number of sample, you run into the danger of overfitting. In such cases, it does make sense to use PCA (which can capture a dominant part of data variance; orthogonality isn't an issue) or factor analysis (which can find the true explanatory variables underlying the data) to reduce data dimensionality and then train a regression model with them.

For factor analysis based approaches, see this paper Bayesian Factor Regression Model, and a nonparametric Bayesian version of this model that does not assume that you a priori know the "true" number of relevant factors (or principal components in case of PCA).

I'd add that in many cases, supervised dimensionality reduction (e.g., Fisher Discriminant Analysis) can give improvements over simple PCA or FA based approaches, because you can make use of the label information while doing dimensionality reduction.

Answer 6

you might pull it out if the predicted PC score was extracted from different variables, or cases, than the predictor PC scores. if that be the case predicted and predictor will not be orthogonal, or at least they need not be, correlation is, of course, not guaranteed.