15
$\begingroup$

A while ago a user on R-help mailing list asked about the soundness of using PCA scores in a regression. The user is trying to use some PC scores to explain variation in another PC (see full discussion here). The answer was that no, this is not sound because PCs are orthogonal to each other.

Can someone explain in a bit more detail why this is so?

| cite | improve this question | |
$\endgroup$
11
$\begingroup$

A principal component is a weighted linear combination of all your factors (X's).

example: PC1 = 0.1X1 + 0.3X2

There will be one component for each factor (though in general a small number are selected).

The components are created such that they have zero correlation (are orthogonal), by design.

Therefore, component PC1 should not explain any variation in component PC2.

You may want to do regression on your Y variable and the PCA representation of your X's, as they will not have multi-collinearity. However, this could be hard to interpret.

If you have more X's than observations, which breaks OLS, you can regress on your components, and simply select a smaller number of the highest variation components.

Principal Component Analysis by Jollife a very in-depth and highly cited book on the subject

This is also good: http://www.statsoft.com/textbook/principal-components-factor-analysis/

| cite | improve this answer | |
$\endgroup$
11
$\begingroup$

Principal components are orthogonal by definition, so any pair of PCs will have zero correlation.

However, PCA can be used in regression if there are a large number of explanatory variables. These can be reduced to a small number of principal components and used as predictors in a regression.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Wouldn't that be FA then? $\endgroup$ – Roman Luštrik Jul 20 '10 at 5:52
  • 3
    $\begingroup$ No. FA is not regression. I am referring to a response variable regressed against the principal components computed from a large number of explanatory variables. The principal components themselves are closely related to the factors of FA. $\endgroup$ – Rob Hyndman Jul 20 '10 at 6:14
  • $\begingroup$ I'm sorry, I should have been more precise in my comment. Your writing that the explanatory variables can be reduced to a small number of PC rang me "factor analysis" bell. $\endgroup$ – Roman Luštrik Jul 20 '10 at 7:32
  • $\begingroup$ In the set with n variables, n PCs can be extracted, but you can decide how many you'd like to keep, e.g. Guttman-Keiser criterion says: keep all PCs that have eigenvalue (variance) larger than 1. So there... $\endgroup$ – aL3xa Aug 2 '10 at 0:43
7
$\begingroup$

Careful... just because the PCs are by construction orthogonal to each other does not mean that there is not a pattern or that one PC can not appear to "explain" something about the other PCs.

Consider 3D data (X,Y,Z) describing a large number of points distributed evenly on the surface of an American football (it is an ellipsoid -- not a sphere -- for those who have never watched American football). Imagine that the football is in an arbitrary configuration so that neither X nor Y nor Z is along the long axis of the football.

Principal components will place PC1 along the long axis of the football, the axis that describes the most variance in the data.

For any point in the PC1 dimension along the long axis of the football, the planar slice represented by PC2 and PC3 should describe a circle and the radius of this circular slice depends on the PC1 dimension. It is true that regressions of PC2 or PC3 on PC1 should give a zero coefficient globally, but not over smaller sections of the football.... and it is clear that a 2D graph of PC1 and PC2 would show an "interesting" limiting boundary that is two-valued, nonlinear, and symmetric.

| cite | improve this answer | |
$\endgroup$
3
$\begingroup$

If your data is high dimensional and noisy, and you don't have a large number of sample, you run into the danger of overfitting. In such cases, it does make sense to use PCA (which can capture a dominant part of data variance; orthogonality isn't an issue) or factor analysis (which can find the true explanatory variables underlying the data) to reduce data dimensionality and then train a regression model with them.

For factor analysis based approaches, see this paper Bayesian Factor Regression Model, and a nonparametric Bayesian version of this model that does not assume that you a priori know the "true" number of relevant factors (or principal components in case of PCA).

I'd add that in many cases, supervised dimensionality reduction (e.g., Fisher Discriminant Analysis) can give improvements over simple PCA or FA based approaches, because you can make use of the label information while doing dimensionality reduction.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

you might pull it out if the predicted PC score was extracted from different variables, or cases, than the predictor PC scores. if that be the case predicted and predictor will not be orthogonal, or at least they need not be, correlation is, of course, not guaranteed.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.