Is it actually fine to perform unsupervised feature selection before cross-validation?

Question

In The Elements of Statistical Learning, I've found the following statement:

There is one qualification: initial unsupervised screening steps can be done before samples are left out. For example, we could select the 1000 predictors with highest variance across all 50 samples, before starting cross-validation. Since this filtering does not involve the class labels, it does not give the predictors an unfair advantage.

Is this actually valid? I mean, by filtering attributes beforehand, we are not imitating the training data/new data environment - so does this matter that the filtering we are performing is not supervised? Isn't it better to actually do all preprocessing steps within cross-validation process? If that's not the case, then it means that all the unsupervised preprocessing can be performed beforehand, including feature normalization/PCA, etc. But by doing these on the whole training set, we are actually leaking some data to the training set. I can agree that with relatively stable dataset, these differences should most likely very tiny - but it does not mean they don't exist, right? What's the correct way to think about this?

Answer 1

As a stage done before cross validation, unsupervised feature selection is somewhat similar to feature normalization:

1. From the point of view of a specific fold in the cross validation, the train data peeked at the test data (albeit only in the independent variables).

2. This peeking is relatively mild.

Feature normalization before cross validation was discussed in this question. Quoting the answer there by Dikran Marsupial

Cross-validation is best viewed as a method to estimate the performance of a statistical procedure, rather than a statistical model. Thus in order to get an unbiased performance estimate, you need to repeat every element of that procedure separately in each fold of the cross-validation, which would include normalisation.

So if you can spare the resources, the best thing would be to have each cross-validation fold do any data-dependent processing from scratch.

However, as the answers to that question say, in practice, reversing the order probably wouldn't change things much. There certainly isn't the substantial unfair advantage that $y$-dependent feature selection exhibits. IMHO, that's the interpretation of the quote from Elements Of Statistical Learning.

Answer 2

I beg to differ in this question with @AmiTavory's opinion as well as with the Elements of Statistical Learning.

Coming from an applied field with very low sample sizes, I have the experience that also unsupervised pre-processing steps can introduce severe bias.

In my field that would be most frequently PCA for dimensionality reduction before a classifier is trained. While I cannot show the data here, I've seen PCA + (cross validated LDA) vs. cross validated (PCA + LDA) underestimating the error rate by about an order of magnitude. (This is usually an indicator that that the PCA is not stable.)

As for the "unfair advantage" argumentation of the Elements, if variance of taining + test cases is examined, we end up with features that work well with both the training and test cases. Thus, we create a self-fulfilling prophecy here which is the cause of the overoptimistic bias. This bias is low if you have reasonably comfortable sample sizes.

So I recommend an approach that is slightly more conservative than the Elements:

• preprocessing calculations that consider more than one case need to be included in the validation: i.e. they are calculated on the respective training set only (and then applied to the test data)
• preprocessing steps that consider each case on its own (I'm spectroscopist: examples would be baseline correction and intensity normalization, which is a row-wise normalization) may be pulled out of the cross validation as long as they are before the first step that calculates for multiple cases.

That being said, also cross valiation is only a short-cut for doing a proper validation study. Thus, you may argue with practicality:

• You could check whether the pre-processing in question yields stable results (you can do that e.g. by cross validation). If you find it perfectly stable already with lower sample sizes, IMHO you may argue that not much bias will be introduced by pulling it out of the cross validation.

• However, to cite a previous supervisor: Calculation time is no scientific argument.
  I often go for a "sneak preview" of few folds and few iterations for the cross validation to make sure all code (including the summary/graphs of the results) and then leave it over night or over weekend or so on the server for a more fine-grained cross validation.