Update: I raised this issue with Professor Hastie in a brief email correspondence. To summarize, his feeling is that:
This is an interesting topic which we have explored with our students in various ways, but I personally am still pretty happy with the statement in ESL, although not as adamant as I was at the time we wrote it.
To supplement cbeleites' answer, here is a simulation experiment which provides evidence for this claim:
Coming from an applied field with very low sample sizes, I have the experience that also unsupervised pre-processing steps can introduce severe bias. In my field that would be most frequently PCA for dimensionality reduction before a classifier is trained.
Methodology
The experiment procedure:
Generate a sample of features and labels according to a linear model. The rank of these features is an input.
Split the sample into 3 subsamples:
train
: 100 observations for supervised training
test
: an inputted number of observations for testing (this will range from 25 to 450)
extra
: same number of observations as the test subsample, for unsupervised training.
Fit a PCA on test
features, apply it to train
features and test
features, train a linear model on PCA'd train
features and labels, and compute the RMSE of this model on PCA'd test
features and labels. Call this RMSE $\text{error}_{\text{test}}$. The number of PCA components is less than the effective rank of the features generated in (1), as is usually the case in practice.
Same as (3) except that the PCA is fit on extra
features. Call this RMSE $\text{error}_{\text{extra}}$.
$\text{error}_{\text{test}}$ and $\text{error}_{\text{extra}}$ are paired, as the supervised training and test sets are identical. The only difference is the source (but not the size) of unsupervised training data. The sources of randomness are the particular splits which determine the subsamples, so the experiment procedure will be repeated 300 times.
$\text{error}_{\text{extra}}$ is clearly an unbiased estimator of out-of-sample RMSE, as it's never trained on features or labels which depend on test set features or labels. It's unclear whether $\text{error}_{\text{test}}$ is unbiased, as it's trained on test set features (but not labels). If The Elements of Statistical Learning1 is right—
initial unsupervised screening steps can be done before samples are left out . . . Since this filtering does not involve the class labels, it does not give the predictors an unfair advantage.
—then $\text{E}[\text{error}_{\text{extra}} - \text{error}_{\text{test}}] = 0$, i.e., there is no underestimation of out-of-sample RMSE despite (unsupervised) training on test
.
Results
Code to reproduce the results of this experiment is here.
The first two plots demonstrate that the degree of underestimation may depend on the sample size: as the sample size increases, there's less underestimation.


Interestingly, there's no evidence of underestimation if the features are full rank (note that the scale of the y-axis is 1e-16
).

It's tempting to conclude that the amount of underestimation depends on how much the unsupervised training procedure helps. But my question here provides some empirical evidence against that.
References
- Hastie, Trevor, et al. The elements of statistical learning: data mining, inference, and prediction. Vol. 2. New York: springer, 2009.