25
$\begingroup$

Suppose I am running a regression $Y \sim X$. Why by selecting top $k$ principle components of $X$, does the model retain its predictive power on $Y$?

I understand that from dimensionality-reduction/feature-selection point of view, if $v_1, v_2, ... v_k$ are the eigenvectors of covariance matrix of $X$ with top $k$ eigenvalues, then $Xv_1, Xv_2 ... Xv_k$ are top $k$ principal components with maximum variances. We can thereby reduce the number of features to $k$ and retain most of the predictive power, as I understand it.

But why do top $k$ components retain the predictive power on $Y$?

If we talk about a general OLS $Y \sim Z$, there is no reason to suggest that if feature $Z_i$ has maximum variance, then $Z_i$ has the most predictive power on $Y$.

Update after seeing comments: I guess I have seen tons of examples of using PCA for dimensionality reduction. I have been assuming that means the dimensions we are left with have the most predictive power. Otherwise what's the point of dimensionality reduction?

$\endgroup$
  • 3
    $\begingroup$ You are correct: there is no mathematical reason to suppose the top $K$ PCs of $X$ have any predictive power--just as there is no mathematical reason generally to suppose that any set of covariates $X$ has any relationship at all to a given $Y$. It seems you might be referring to some statement you have encountered: exactly what does it say and who said it? $\endgroup$ – whuber Mar 15 '15 at 20:56
  • $\begingroup$ @whuber I guess I have seem tons of examples of using PCA for dimension reduction. I have been assuming that means the dimensions we are left with have the most predictive powers. Otherwise that's the point of reducing dimensions? $\endgroup$ – CodeNoob Mar 15 '15 at 20:59
40
$\begingroup$

Indeed, there is no guarantee that top principal components (PCs) have more predictive power than the low-variance ones.

Real-world examples can be found where this is not the case, and it is easy to construct an artificial example where e.g. only the smallest PC has any relation to $y$ at all.

This topic was discussed a lot on our forum, and in the (unfortunate) absence of one clearly canonical thread, I can only give several links that together provide various real life as well as artificial examples:

And the same topic, but in the context of classification:


However, in practice, top PCs often do often have more predictive power than the low-variance ones, and moreover, using only top PCs can yield better predictive power than using all PCs.

In situations with a lot of predictors $p$ and relatively few data points $n$ (e.g. when $p \approx n$ or even $p>n$), ordinary regression will overfit and needs to be regularized. Principal component regression (PCR) can be seen as one way to regularize the regression and will tend to give superior results. Moreover, it is closely related to ridge regression, which is a standard way of shrinkage regularization. Whereas using ridge regression is usually a better idea, PCR will often behave reasonably well. See Why does shrinkage work? for the general discussion about bias-variance tradeoff and about how shrinkage can be beneficial.

In a way, one can say that both ridge regression and PCR assume that most information about $y$ is contained in the large PCs of $X$, and this assumption is often warranted.

See the later answer by @cbeleites (+1) for some discussion about why this assumption is often warranted (and also this newer thread: Is dimensionality reduction almost always useful for classification? for some further comments).

Hastie et al. in The Elements of Statistical Learning (section 3.4.1) comment on this in the context of ridge regression:

[T]he small singular values [...] correspond to directions in the column space of $\mathbf X$ having small variance, and ridge regression shrinks these directions the most. [...] Ridge regression protects against the potentially high variance of gradients estimated in the short directions. The implicit assumption is that the response will tend to vary most in the directions of high variance of the inputs. This is often a reasonable assumption, since predictors are often chosen for study because they vary with the response variable, but need not hold in general.

See my answers in the following threads for details:


Bottom line

For high-dimensional problems, pre-processing with PCA (meaning reducing dimensionality and keeping only top PCs) can be seen as one way of regularization and will often improve the results of any subsequent analysis, be it a regression or a classification method. But there is no guarantee that this will work, and there are often better regularization approaches.

$\endgroup$
  • $\begingroup$ Thank you for collecting the references in your answer. Here is another one recent. There is an answer with further links. $\endgroup$ – ttnphns Mar 16 '15 at 16:12
  • $\begingroup$ Thanks, @ttnphns! I have not seen that post, because it did not have a [pca] tag (I am closely following only a handful of specific tags). Actually, I am rather unhappy that there is a loose collection of 5-10 closely related threads, with no question and no answer being really perfect and no real duplicates between them. I would prefer to have one canonical thread that could be used for future references... $\endgroup$ – amoeba Mar 16 '15 at 16:18
  • $\begingroup$ I've added the tag to that question. The "perfect" encyclodedic answer on that interesting theme is awaiting its author. :-) You might decide to become a one. $\endgroup$ – ttnphns Mar 16 '15 at 16:23
  • $\begingroup$ Also relevant: onestop's answer to stats.stackexchange.com/questions/3561/… $\endgroup$ – kjetil b halvorsen Mar 20 '15 at 12:17
11
$\begingroup$

In addition to the answers that already focus on the mathematical properties, I'd like to comment from an experimental point of view.

Summary: data generation processes are often optimized in a way that makes the data suitable for principal component (PCR) or partial least squares (PLS) regression.


I'm analytical chemist. When I designing an experiment/method to measure (regression or classification) something, I use my knowledge about application and available instruments to get data that carries a good signal to noise ratio with respect to the task at hand. That means, the data I generate is designed to have large covariance with the property of interest.
This leads to a variance structure where the interesting variance is large, and the later PCs will carry the (small) noise only.

I'd also prefer methods that yield redundant information about the task at hand, in order to have more robust or more precise results. PCA concentrates redundant measurement channels into one PC, which then carries much variance and is therefore one of the first PCs.

If there are known confounders that will lead to large variance that is not correlated with the property of interest, I'll usually try to correct for these as much as possible during the preprocessing of the data: in many cases these confounders are of a known physical or chemical nature, and this knowledge suggests appropriate ways to correct for the confounders. E.g. I measure Raman spectra under the microscope. Their intensity depends on the intensity of the laser light as well as on how well I can focus the microscope. Both lead to changes that can be corrected by normalizing e.g. to a signal that is known to be constant.
Thus, large contributors of variance that does not contribute to the solution may have been eliminated before the data enters PCA, leaving mostly meaningful variance in the first PCs.


Last but not least, there's a bit of a self-fulfilling prophecy here: Obviously PCR is done with data where the assumption that the information carrying variance is large does make sense. If e.g. I think that there could be important confounders that I don't know how to correct for, I'd immediately go for PLS which is better at ignoring large contributions that do not help with the prediction task.

$\endgroup$
  • $\begingroup$ +1. This is an excellent addition, thanks for joining the discussion. $\endgroup$ – amoeba Mar 21 '15 at 14:20
  • $\begingroup$ @amoeba: thanks for the kind words. As always, also your answer is very thorough. I do in fact rely on you taking care of [pca]! $\endgroup$ – cbeleites Mar 21 '15 at 15:16
6
$\begingroup$

PCA is sometimes used to correct problems caused by collinear variables so that most of the variation in the X space is captured by the K principal components.

But this mathematical problem is of course not the same as capturing most of variation both in X, Y space in such way that unexplained variation is as small as possible.

Partial least squares tries to do this in the latter sense:

http://en.wikipedia.org/wiki/Partial_least_squares_regression

$\endgroup$
3
$\begingroup$

As other has pointed out, there is no direct link between top k eigenvectors and the predictive power. By picking the top and using them as basis, you are retaining some top energy (or variance along those axis).

It can be that the axis explaining the most variance are actually useful for prediction but in general this is not the case.

$\endgroup$
  • $\begingroup$ When you say "in general", do you mean in general in practice or in general in theory? $\endgroup$ – amoeba Mar 16 '15 at 15:59
  • $\begingroup$ @amoeba In general because it is easy to construct a dataset where projecting the data on top k maximum variance axis is not predictive/discriminative. $\endgroup$ – Vladislavs Dovgalecs Mar 16 '15 at 18:41
-1
$\begingroup$

Let me offer one simple explanation.

PCA amounts to removing certain features intuitively. This decreases chances of over-fitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.