12
$\begingroup$

In Statistical Methods in the Atmospheric Sciences, Daniel Wilks notes that multiple linear regression can lead to problems if there are very strong intercorrelations among the predictors (3rd edition, page 559-560):

A pathology that can occur in multiple linear regression is that a set of predictor variables having strong mutual correlations can result in the calculation of an unstable regression relationship.

(...)

He then introduces principal component regression:

An approach to remedying this problem is to first transform the predictors to their principal components, the correlations among which are zero.

So far so good. But next, he makes some statements that he does not explain (or at least not in sufficient detail for me to understand):

If all the principal components are retained in a principal component regression, then nothing is gained over the conventional least-squares fit to the full predictor set.

(..) and:

It is possible to reexpress the principal-component regression in terms of the original predictors, but the result will in general involve all the original predictor variables even if only one or a few principal component predictors have been used. This reconstituted regression will be biased, although often the variance is much smaller, resulting in a smaller MSE overall.

I don't understand these two points.

Of course, if all the principal components are retained, we use the same information as when we were using the predictors in their original space. However, the problem of mutual correlations is removed by working in principal component space. We may still have overfitting, but is that the only problem? Why is nothing gained?

Secondly, even if we do truncate the principal components (perhaps for noise reduction and/or to prevent overfitting), why and how does this lead to a biased reconstituted regression? Biased in what way?


Book source: Daniel S. Wilks, Statistical Methods in the Atmospheric Sciences, Third edition, 2011. International Geophysics Series Volume 100, Academic Press.

$\endgroup$
  • 4
    $\begingroup$ (+1) In the second quotation, "will be biased" does not follow logically: better would be a milder statement such as "likely to be biased." I suspect the reasoning behind it may be something along the lines of "because PCR imposes linear relations among the parameter estimates, those estimates will tend to differ from OLS estimates; and because OLS estimates are unbiased, that means PCR estimates will be biased." Intuitively it's a good heuristic but it's not quite correct. $\endgroup$ – whuber Dec 15 '15 at 22:33
  • $\begingroup$ could it be said that " PCR will be biased" if (a) data points do not occupy a linear lower or equal dimensional manifold than chosen number of PCs and (b) data points are not perfectly uncorrelated? or how? $\endgroup$ – Soren Havelund Welling Dec 15 '15 at 22:47
15
$\begingroup$

What happens when all PCs are used?

If all PCs are used, then the resulting regression coefficients will be identical to the ones obtained with OLS regression, and so this procedure should better not be called "principal component regression". It is standard regression, only performed in a roundabout way.

You are asking how it is possible that nothing is gained, given that after PCA the predictors become orthogonal. The devil hides in the back-transformation of the regression coefficients from the PCA space to the original space. What you need to know is that the variance of the estimated regression coefficients inversely depends on the covariance matrix of the predictors. The PCA-transformed predictors, let's call them $Z$, have diagonal covariance matrix (because they are uncorrelated). So all regression coefficients for $Z$ are also uncorrelated; the ones corresponding to the high-variance PCs have low variance (i.e. are estimated reliably) and the ones corresponding to the low-variance PCs have high variance (i.e. are estimated unreliably). When these coefficients are back-transformed to the original predictors $X$, each of the predictors $X_i$ will get some portion of the unreliable estimates, and so all coefficients can become unreliable.

So nothing is gained.

What happens when only few PCs are used?

When not all the PCs are retained in PCR, then the resulting solution $\hat \beta_\mathrm{PCR}$ will generally not be equal to the standard ordinary least squares solution $\hat \beta_\mathrm{OLS}$. It is a standard result that OLS solution is unbiased: see Gauss-Markov theorem. "Unbiased" means that $\hat \beta$ is correct on average, even though it can be very noisy. Since PCR solution differs from it, it will be biased, meaning that it will be incorrect on average. However, it often happens that it is substantially less noisy, leading to the overall more accurate predictions.

This is an example of the bias-variance trade-off. See Why does shrinkage work? for some further general discussion.

In the comments, @whuber pointed out that the PCR solution does not have to differ from the OLS one and hence does not have to be biased. Indeed, if the dependent variable $y$ is uncorrelated (in population, not in sample) with all the low-variance PCs that are not included in the PCR model, then dropping these PCs will not influence the unbiasedness. This, however, is unlikely to be the case in practice: PCA is conducted without taking $y$ into account so it stands to reason that $y$ will tend to be somewhat correlated with all the PCs.

Why using high-variance PCs is a good idea at all?

This was not part of the question, but you might be interested in the following thread for the further reading: How can top principal components retain the predictive power on a dependent variable (or even lead to better predictions)?

$\endgroup$
  • $\begingroup$ The comments in your last paragraph appear to confound the conditional distribution of $Y$ (on the regressors) with the values of $Y$ in the dataset. $\endgroup$ – whuber Dec 16 '15 at 0:16
  • $\begingroup$ @whuber, Indeed. I have rewritten that paragraph, hopefully it makes more sense now. Thanks. $\endgroup$ – amoeba Dec 16 '15 at 0:28
  • $\begingroup$ Hmm, right. Biased essentially means some points are more equal than others, which is exactly what we want if we want to reduce the influence of noise and outliers (which I'm not yet sure if PCA is the best tool for). $\endgroup$ – gerrit Dec 16 '15 at 12:22
  • $\begingroup$ @gerrit You write as if bias were tantamount to weighting, but they are separate things. Bias in this context refers to any difference between the expected values of the coefficient estimates and their true values. $\endgroup$ – whuber Dec 16 '15 at 23:53
  • 1
    $\begingroup$ The model supposes the responses are random variables. This makes the estimated coefficients--using any procedure--random variables also. Their expected values are defined as for any random variable. By definition, bias is the difference between the expected value and the true value. OLS coefficient estimates have biases of zero. The bias of some other procedure still could be zero. The logic of the quotation is that a procedure which is linear, like OLS, but imposes relations among the coefficients, will necessarily be biased. That conclusion is true in many cases, but not all. $\endgroup$ – whuber Dec 17 '15 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.