Skip to main content
1 of 5
amoeba
  • 107.2k
  • 36
  • 321
  • 346

Indeed, there is no guarantee that top principal components (PCs) have more predictive power than the small ones.

Real-world examples can be found where this is not the case, and it easy to construct an artificial example where e.g. only the smallest PC has any relation to $y$ at all.

This topic was discussed a lot on our forum, and in the (unfortunate) absence of one clearly canonical thread, I can only give several links that together provide an overview and some examples:

And the same topic, but in the context of classification:

However, it would be misleading to stop here without adding the following. In situations with a lot of predictors $p$ and relatively few data points $n$ (e.g. when $p \approx n$ or even $p>n$), ordinary regression will overfit and needs to be regularized. Principal component regression (PCR) can be seen as one way to regularize the regression and will tend to give superior results. Moreover, it is closely connected to ridge regression, which is a standard way of regularization (based on shrinkage). Whereas using ridge regression is usually a better idea, PCR will often behave reasonably well.

In a way, one can say that both ridge regression and PCR assume that most information about $y$ is contained in the large PCs of $X$, and this assumption is often warranted.

See my answers in the following threads for details:

amoeba
  • 107.2k
  • 36
  • 321
  • 346