In Statistical Methods in the Atmospheric Sciences, Daniel Wilks notes that multiple linear regression can lead to problems if there are very strong intercorrelations among the predictors (3rd edition, page 559-560):
A pathology that can occur in multiple linear regression is that a set of predictor variables having strong mutual correlations can result in the calculation of an unstable regression relationship.
He then introduces principal component regression:
An approach to remedying this problem is to first transform the predictors to their principal components, the correlations among which are zero.
So far so good. But next, he makes some statements that he does not explain (or at least not in sufficient detail for me to understand):
If all the principal components are retained in a principal component regression, then nothing is gained over the conventional least-squares fit to the full predictor set.
It is possible to reexpress the principal-component regression in terms of the original predictors, but the result will in general involve all the original predictor variables even if only one or a few principal component predictors have been used. This reconstituted regression will be biased, although often the variance is much smaller, resulting in a smaller MSE overall.
I don't understand these two points.
Of course, if all the principal components are retained, we use the same information as when we were using the predictors in their original space. However, the problem of mutual correlations is removed by working in principal component space. We may still have overfitting, but is that the only problem? Why is nothing gained?
Secondly, even if we do truncate the principal components (perhaps for noise reduction and/or to prevent overfitting), why and how does this lead to a biased reconstituted regression? Biased in what way?
Book source: Daniel S. Wilks, Statistical Methods in the Atmospheric Sciences, Third edition, 2011. International Geophysics Series Volume 100, Academic Press.