# PCA scores in multiple regression

A while ago a user on R-help mailing list asked about the soundness of using PCA scores in a regression. The user is trying to use some PC scores to explain variation in another PC (see full discussion here). The answer was that no, this is not sound because PCs are orthogonal to each other. Can someone explain in a bit more detail why this is so?

-
Why did you put r tag and what do you mean by "why this is so"? PC's are not correlated, i.e. they're orthogonal, additive, you cannot predict one PC with the another. Are you looking for a formula? –  aL3xa Aug 2 '10 at 0:38
I was wondering about the principles behind the logic (in my quest to understand PCA). I used R tag because R people might read this and maybe show R examples. :) –  Roman Luštrik Aug 2 '10 at 6:13
Oh, why didn't you say so? Have you seen statmethods.net/advstats/factor.html –  aL3xa Aug 2 '10 at 11:01

A principal component is a weighted linear combination of all your factors (X's).

example: PC1 = 0.1X1 + 0.3X2

There will be one component for each factor (though in general a small number are selected).

The components are created such that they have zero correlation (are orthogonal), by design.

Therefore, component PC1 should not explain any variation in component PC2.

You may want to do regression on your Y variable and the PCA representation of your X's, as they will not have multi-collinearity. However, this could be hard to interpret.

If you have more X's than observations, which breaks OLS, you can regress on your components, and simply select a smaller number of the highest variation components.

Principal Component Analysis by Jollife a very in-depth and highly cited book on the subject

This is also good: http://www.statsoft.com/textbook/principal-components-factor-analysis/

-

Principal components are orthogonal by definition, so any pair of PCs will have zero correlation.

However, PCA can be used in regression if there are a large number of explanatory variables. These can be reduced to a small number of principal components and used as predictors in a regression.

-
Wouldn't that be FA then? –  Roman Luštrik Jul 20 '10 at 5:52
No. FA is not regression. I am referring to a response variable regressed against the principal components computed from a large number of explanatory variables. The principal components themselves are closely related to the factors of FA. –  Rob Hyndman Jul 20 '10 at 6:14
I'm sorry, I should have been more precise in my comment. Your writing that the explanatory variables can be reduced to a small number of PC rang me "factor analysis" bell. –  Roman Luštrik Jul 20 '10 at 7:32
In the set with n variables, n PCs can be extracted, but you can decide how many you'd like to keep, e.g. Guttman-Keiser criterion says: keep all PCs that have eigenvalue (variance) larger than 1. So there... –  aL3xa Aug 2 '10 at 0:43

Careful... just because the PCs are by construction orthogonal to each other does not mean that there is not a pattern or that one PC can not appear to "explain" something about the other PCs.

Consider 3D data (X,Y,Z) describing a large number of points distributed evenly on the surface of an American football (it is an ellipsoid -- not a sphere -- for those who have never watched American football). Imagine that the football is in an arbitrary configuration so that neither X nor Y nor Z is along the long axis of the football.

Principal components will place PC1 along the long axis of the football, the axis that describes the most variance in the data.

For any point in the PC1 dimension along the long axis of the football, the planar slice represented by PC2 and PC3 should describe a circle and the radius of this circular slice depends on the PC1 dimension. It is true that regressions of PC2 or PC3 on PC1 should give a zero coefficient globally, but not over smaller sections of the football.... and it is clear that a 2D graph of PC1 and PC2 would show an "interesting" limiting boundary that is two-valued, nonlinear, and symmetric.

-
+1 Very nice intuitive explanation –  David Sep 6 '12 at 4:41

If your data is high dimensional and noisy, and you don't have a large number of sample, you run into the danger of overfitting. In such cases, it does make sense to use PCA (which can capture a dominant part of data variance; orthogonality isn't an issue) or factor analysis (which can find the true explanatory variables underlying the data) to reduce data dimensionality and then train a regression model with them.

For factor analysis based approaches, see this paper Bayesian Factor Regression Model, and a nonparametric Bayesian version of this model that does not assume that you a priori know the "true" number of relevant factors (or principal components in case of PCA).

I'd add that in many cases, supervised dimensionality reduction (e.g., Fisher Discriminant Analysis) can give improvements over simple PCA or FA based approaches, because you can make use of the label information while doing dimensionality reduction.

-