In The Elements of Statistical Learning, I've found the following statement:
There is one qualification: initial unsupervised screening steps can be done before samples are left out. For example, we could select the 1000 predictors with highest variance across all 50 samples, before starting cross-validation. Since this filtering does not involve the class labels, it does not give the predictors an unfair advantage.
Is this actually valid? I mean, by filtering attributes beforehand, we are not imitating the training data/new data environment - so does this matter that the filtering we are performing is not supervised? Isn't it better to actually do all preprocessing steps within cross-validation process? If that's not the case, then it means that all the unsupervised preprocessing can be performed beforehand, including feature normalization/PCA, etc. But by doing these on the whole training set, we are actually leaking some data to the training set. I can agree that with relatively stable dataset, these differences should most likely very tiny - but it does not mean they don't exist, right? What's the correct way to think about this?