Return to Answer

• Low variance components in PCA, are they really just noise? Is there any way to test for it?
• Examples of PCA where PCs with low variance are "useful"
• How can a later principal component be significant predictor in a regression, when an earlier PC is not?
• How to use principal components analysis to select variables for regression?

• What can cause PCA to worsen results of a classifier?
• The first principal component does not separate classes, but other PCs do; how is that possible?

In situations with a lot of predictors $p$ and relatively few data points $n$ (e.g. when $p \approx n$ or even $p>n$), ordinary regression will overfit and needs to be regularized. Principal component regression (PCR) can be seen as one way to regularize the regression and will tend to give superior results. Moreover, it is closely related to ridge regression, which is a standard way of shrinkage regularization. Whereas using ridge regression is usually a better idea, PCR will often behave reasonably well. See Why does shrinkage work? for the general discussion about bias-variance tradeoff and about how shrinkage can be beneficial.

See the later answer by @cbeleites (+1) for some discussion about why this assumption is often warranted (and also this newer thread: Is dimensionality reduction almost always useful for classification? for some further comments).

• What is the advantage of reducing dimensionality of predictors for the purposes of regression?
• Relationship between ridge regression and PCA regression
• Does it make sense to combine PCA and LDA?

• Low variance components in PCA, are they really just noise? Is there any way to test for it?
• Examples of PCA where PCs with low variance are "useful"
• How can a later principal component be significant predictor in a regression, when an earlier PC is not?
• How to use principal components analysis to select variables for regression?

• What can cause PCA to worsen results of a classifier?
• The first principal component does not separate classes, but other PCs do; how is that possible?

In situations with a lot of predictors $p$ and relatively few data points $n$ (e.g. when $p \approx n$ or even $p>n$), ordinary regression will overfit and needs to be regularized. Principal component regression (PCR) can be seen as one way to regularize the regression and will tend to give superior results. Moreover, it is closely related to ridge regression, which is a standard way of shrinkage regularization. Whereas using ridge regression is usually a better idea, PCR will often behave reasonably well. See Why does shrinkage work? for the general discussion about bias-variance tradeoff and about how shrinkage can be beneficial.

See the later answer by @cbeleites (+1) for some discussion about why this assumption is often warranted (and also this newer thread: Is dimensionality reduction almost always useful for classification? for some further comments).

• What is the advantage of reducing dimensionality of predictors for the purposes of regression?
• Relationship between ridge regression and PCA regression
• Does it make sense to combine PCA and LDA?

Indeed, there is no guarantee that top principal components (PCs) have more predictive power than the small ones.

However, it would be misleading to stop here without adding the following. In situations with a lot of predictors $p$ and relatively few data points $n$ (e.g. when $p \approx n$ or even $p>n$), ordinary regression will overfit and needs to be regularized. Principal component regression (PCR) can be seen as one way to regularize the regression and will tend to give superior results. Moreover, it is closely connected to ridge regression, which is a standard way of regularization (based on shrinkage). Whereas using ridge regression is usually a better idea, PCR will often behave reasonably well.

In a way, one can say that both ridge regression and PCR assume that most information about $y$ is contained in the large PCs of $X$, and this assumption is often warranted.

Indeed, there is no guarantee that top principal components (PCs) have more predictive power than the low-variance ones.

However, in practice, top PCs often do often have more predictive power than the low-variance ones, and moreover, using only top PCs can yield better predictive power than using all PCs.

In situations with a lot of predictors $p$ and relatively few data points $n$ (e.g. when $p \approx n$ or even $p>n$), ordinary regression will overfit and needs to be regularized. Principal component regression (PCR) can be seen as one way to regularize the regression and will tend to give superior results. Moreover, it is closely related to ridge regression, which is a standard way of shrinkage regularization. Whereas using ridge regression is usually a better idea, PCR will often behave reasonably well. See Why does shrinkage work? for the general discussion about bias-variance tradeoff and about how shrinkage can be beneficial.

In a way, one can say that both ridge regression and PCR assume that most information about $y$ is contained in the large PCs of $X$, and this assumption is often warranted.

Bottom line

For high-dimensional problems, pre-processing with PCA (meaning reducing dimensionality and keeping only top PCs) can be seen as one way of regularization and will often improve the results of any subsequent analysis, be it a regression or a classification method. But there is no guarantee that this will work, and there are often better regularization approaches.

Indeed, there is no guarantee that top principal components (PCs) have more predictive power than the small ones.

Real-world examples can be found where this is not the case, and it easy to construct an artificial example where e.g. only the smallest PC has any relation to $y$ at all.

This topic was discussed a lot on our forum, and in the (unfortunate) absence of one clearly canonical thread, I can only give several links that together provide an overview and some examples:

• Low variance components in PCA, are they really just noise? Is there any way to test for it?
• Examples of PCA where PCs with low variance are "useful"
• How can a later principal component be significant predictor in a regression, when an earlier PC is not?
• How to use principal components analysis to select variables for regression?

And the same topic, but in the context of classification:

• What can cause PCA to worsen results of a classifier?
• The first principal component does not separate classes, but other PCs do; how is that possible?
• Does it make sense to combine PCA and LDA?

However, it would be misleading to stop here without adding the following. In situations with a lot of predictors $p$ and relatively few data points $n$ (e.g. when $p \approx n$ or even $p>n$), ordinary regression will overfit and needs to be regularized. Principal component regression (PCR) can be seen as one way to regularize the regression and will tend to give superior results. Moreover, it is closely connected to ridge regression, which is a standard way of regularization (based on shrinkage). Whereas using ridge regression is usually a better idea, PCR will often behave reasonably well.

In a way, one can say that both ridge regression and PCR assume that most information about $y$ is contained in the large PCs of $X$, and this assumption is often warranted.

Hastie et al. in The Elements of Statistical Learning (section 3.4.1) comment on this in the context of ridge regression:

[T]he small singular values [...] correspond to directions in the column space of $\mathbf X$ having small variance, and ridge regression shrinks these directions the most. [...] Ridge regression protects against the potentially high variance of gradients estimated in the short directions. The implicit assumption is that the response will tend to vary most in the directions of high variance of the inputs. This is often a reasonable assumption, since predictors are often chosen for study because they vary with the response variable, but need not hold in general.

See my answers in the following threads for details:

• What is the advantage of reducing dimensionality of predictors for the purposes of regression?
• Relationship between ridge regression and PCA regression

Indeed, there is no guarantee that top principal components (PCs) have more predictive power than the small ones.

Real-world examples can be found where this is not the case, and it is easy to construct an artificial example where e.g. only the smallest PC has any relation to $y$ at all.

This topic was discussed a lot on our forum, and in the (unfortunate) absence of one clearly canonical thread, I can only give several links that together provide various real life as well as artificial examples:

• Low variance components in PCA, are they really just noise? Is there any way to test for it?
• Examples of PCA where PCs with low variance are "useful"
• How can a later principal component be significant predictor in a regression, when an earlier PC is not?
• How to use principal components analysis to select variables for regression?

And the same topic, but in the context of classification:

• What can cause PCA to worsen results of a classifier?
• The first principal component does not separate classes, but other PCs do; how is that possible?

However, it would be misleading to stop here without adding the following. In situations with a lot of predictors $p$ and relatively few data points $n$ (e.g. when $p \approx n$ or even $p>n$), ordinary regression will overfit and needs to be regularized. Principal component regression (PCR) can be seen as one way to regularize the regression and will tend to give superior results. Moreover, it is closely connected to ridge regression, which is a standard way of regularization (based on shrinkage). Whereas using ridge regression is usually a better idea, PCR will often behave reasonably well.

In a way, one can say that both ridge regression and PCR assume that most information about $y$ is contained in the large PCs of $X$, and this assumption is often warranted.

See the later answer by @cbeleites (+1) for some discussion about why this assumption is often warranted (and also this newer thread: Is dimensionality reduction almost always useful for classification? for some further comments).

Hastie et al. in The Elements of Statistical Learning (section 3.4.1) comment on this in the context of ridge regression:

[T]he small singular values [...] correspond to directions in the column space of $\mathbf X$ having small variance, and ridge regression shrinks these directions the most. [...] Ridge regression protects against the potentially high variance of gradients estimated in the short directions. The implicit assumption is that the response will tend to vary most in the directions of high variance of the inputs. This is often a reasonable assumption, since predictors are often chosen for study because they vary with the response variable, but need not hold in general.

See my answers in the following threads for details:

• What is the advantage of reducing dimensionality of predictors for the purposes of regression?
• Relationship between ridge regression and PCA regression
• Does it make sense to combine PCA and LDA?