Is it actually fine to perform unsupervised feature selection before cross-validation?

Question

In The Elements of Statistical Learning, I've found the following statement:

There is one qualification: initial unsupervised screening steps can be done before samples are left out. For example, we could select the 1000 predictors with highest variance across all 50 samples, before starting cross-validation. Since this filtering does not involve the class labels, it does not give the predictors an unfair advantage.

Is this actually valid? I mean, by filtering attributes beforehand, we are not imitating the training data/new data environment - so does this matter that the filtering we are performing is not supervised? Isn't it better to actually do all preprocessing steps within cross-validation process? If that's not the case, then it means that all the unsupervised preprocessing can be performed beforehand, including feature normalization/PCA, etc. But by doing these on the whole training set, we are actually leaking some data to the training set. I can agree that with relatively stable dataset, these differences should most likely very tiny - but it does not mean they don't exist, right? What's the correct way to think about this?

Answer 1

As a stage done before cross validation, unsupervised feature selection is somewhat similar to feature normalization:

1. From the point of view of a specific fold in the cross validation, the train data peeked at the test data (albeit only in the independent variables).

2. This peeking is relatively mild.

Feature normalization before cross validation was discussed in this question. Quoting the answer there by Dikran Marsupial

Cross-validation is best viewed as a method to estimate the performance of a statistical procedure, rather than a statistical model. Thus in order to get an unbiased performance estimate, you need to repeat every element of that procedure separately in each fold of the cross-validation, which would include normalisation.

So if you can spare the resources, the best thing would be to have each cross-validation fold do any data-dependent processing from scratch.

However, as the answers to that question say, in practice, reversing the order probably wouldn't change things much. There certainly isn't the substantial unfair advantage that $y$-dependent feature selection exhibits. IMHO, that's the interpretation of the quote from Elements Of Statistical Learning.

Answer 2

I beg to differ in this question with @AmiTavory's opinion as well as with the Elements of Statistical Learning.

Coming from an applied field with very low sample sizes, I have the experience that also unsupervised pre-processing steps can introduce severe bias.

In my field that would be most frequently PCA for dimensionality reduction before a classifier is trained. While I cannot show the data here, I've seen PCA + (cross validated LDA) vs. cross validated (PCA + LDA) underestimating the error rate by about an order of magnitude. (This is usually an indicator that that the PCA is not stable.)

As for the "unfair advantage" argumentation of the Elements, if variance of taining + test cases is examined, we end up with features that work well with both the training and test cases. Thus, we create a self-fulfilling prophecy here which is the cause of the overoptimistic bias. This bias is low if you have reasonably comfortable sample sizes.

So I recommend an approach that is slightly more conservative than the Elements:

• preprocessing calculations that consider more than one case need to be included in the validation: i.e. they are calculated on the respective training set only (and then applied to the test data)
• preprocessing steps that consider each case on its own (I'm spectroscopist: examples would be baseline correction and intensity normalization, which is a row-wise normalization) may be pulled out of the cross validation as long as they are before the first step that calculates for multiple cases.

That being said, also cross valiation is only a short-cut for doing a proper validation study. Thus, you may argue with practicality:

• You could check whether the pre-processing in question yields stable results (you can do that e.g. by cross validation). If you find it perfectly stable already with lower sample sizes, IMHO you may argue that not much bias will be introduced by pulling it out of the cross validation.

• However, to cite a previous supervisor: Calculation time is no scientific argument.
  I often go for a "sneak preview" of few folds and few iterations for the cross validation to make sure all code (including the summary/graphs of the results) and then leave it over night or over weekend or so on the server for a more fine-grained cross validation.

Answer 3

Update: I raised this issue with Professor Hastie in a brief email correspondence. To summarize, his feeling is that:

This is an interesting topic which we have explored with our students in various ways, but I personally am still pretty happy with the statement in ESL, although not as adamant as I was at the time we wrote it.

---

To supplement cbeleites' answer, here is a simulation experiment which provides evidence for this claim:

Coming from an applied field with very low sample sizes, I have the experience that also unsupervised pre-processing steps can introduce severe bias. In my field that would be most frequently PCA for dimensionality reduction before a classifier is trained.

Methodology

The experiment procedure:

1. Generate a sample of features and labels according to a linear model. The rank of these features is an input.

2. Split the sample into 3 subsamples:

   • train: 100 observations for supervised training
   • test: an inputted number of observations for testing (this will range from 25 to 450)
   • extra: same number of observations as the test subsample, for unsupervised training.

3. Fit a PCA on test features, apply it to train features and test features, train a linear model on PCA'd train features and labels, and compute the RMSE of this model on PCA'd test features and labels. Call this RMSE $\text{error}_{\text{test}}$. The number of PCA components is less than the effective rank of the features generated in (1), as is usually the case in practice.

4. Same as (3) except that the PCA is fit on extra features. Call this RMSE $\text{error}_{\text{extra}}$.

$\text{error}_{\text{test}}$ and $\text{error}_{\text{extra}}$ are paired, as the supervised training and test sets are identical. The only difference is the source (but not the size) of unsupervised training data. The sources of randomness are the particular splits which determine the subsamples, so the experiment procedure will be repeated 300 times.

$\text{error}_{\text{extra}}$ is clearly an unbiased estimator of out-of-sample RMSE, as it's never trained on features or labels which depend on test set features or labels. It's unclear whether $\text{error}_{\text{test}}$ is unbiased, as it's trained on test set features (but not labels). If The Elements of Statistical Learning¹ is right—

initial unsupervised screening steps can be done before samples are left out . . . Since this filtering does not involve the class labels, it does not give the predictors an unfair advantage.

—then $\text{E}[\text{error}_{\text{extra}} - \text{error}_{\text{test}}] = 0$, i.e., there is no underestimation of out-of-sample RMSE despite (unsupervised) training on test.

Results

Code to reproduce the results of this experiment is here.

The first two plots demonstrate that the degree of underestimation may depend on the sample size: as the sample size increases, there's less underestimation.

Interestingly, there's no evidence of underestimation if the features are full rank (note that the scale of the y-axis is 1e-16).

It's tempting to conclude that the amount of underestimation depends on how much the unsupervised training procedure helps. But my question here provides some empirical evidence against that.

References

1. Hastie, Trevor, et al. The elements of statistical learning: data mining, inference, and prediction. Vol. 2. New York: springer, 2009.