Must we do feature selection in cross validation?

Question

It is me again, asking very similar questions with reference to this post, this time, I am a bit confused whether one should perform feature selection before KFold cross validation. I often times see people do one-shot feature selection prior to modelling, and wonder if this is normal?

In one of the top comment, he mentioned:

I don't think that is (quite) what Hastie, et al. are advocating. The general argument is that if feature selection uses the response then it better be included as part of your CV procedure. If you do predictor screening, e.g., by looking at their sample variances and excluding the predictors with small variation, that is ok as a one-shot procedure.

---

Wonder if that means we can do one shot feature selection before cv if we do not take response variable into account?

In fact, if I were to do the feature selection within cross validation, then is there any use of doing extensive EDA on the feature selection?

---

What if there is many multi-collinearity features in my dataset, should I handle it before cross-validation?

---

Edit: I have been reading up extensively on this issue, but it is really cracking my head because of the uncertainty of when it is "ok" to do feature selection/reduction prior to CV. I chanced upon this very short piece of code from scikit-learn in which they used Ward's Method to remove highly correlated features by choosing a cut off point in the clustering process. I wonder if this is something that I can do outside the loop, and if yes, how does one choose the "threshold"?

Answer 1

One of the most challenging elements of this problem is knowing when it's OK to put unsupervised learning steps outside of the CV loop and when they should be fully penalized for by including them inside the loop. Generally speaking, unsupervised learning procedures such as principal components analysis can be unstable, i.e., the loadings of the first principal component will change when computed on a new sample. And unsupervised learning steps to exclude features, such as redundancy analysis and variable clustering, can also be unstable. But their instabilities can either hurt you or help you, i.e., may raise or lower your final $R^2$. So they don't consistently work in your favor. Overfitting in a final predictive discrimination measure such as $R^2$ or pseudo $R^2$ comes from doing things that consistently "work" in your favor, such as doing feature selection using supervised learning, whether manual or automated.

So it's generally OK to keep completely unsupervised learning steps outside the CV loop as pre-processing steps, but sometimes bring them into the loop to make sure your final performance measure doesn't suffer from instability in unsupervised learning.

Answer 2

Cross-validation is a means of estimating the performance of a method for fitting a model, rather than of the model itself, so all steps in fitting the model (including feature selection and optimising the hyper-parameters) need to be performed independently in each fold of the cross-validation procedure. If you don't do this, then you will end up with an optimistically biased performance estimate. See my (with Mrs Marsupial) paper on this topic

GC Cawley and NLC Talbot, "On over-fitting in model selection and subsequent selection bias in performance evaluation", The Journal of Machine Learning Research 11, 2079-2107 (pdf)

I tend to use nested cross-validation to get an unbiased performance estimate, but if you don't need an unbiased performance estimate, just choose between competing methods (that don't have too many degrees of freedom, i.e. not feature selection!) then that often isn't necessary in practice, see Wainer and Cawley

J Wainer and G Cawley, "Nested cross-validation when selecting classifiers is overzealous for most practical applications", Expert Systems with Applications 182, 115 (doi:10.1016/j.eswa.2021.115222)

Once you have that performance estimate, then retrain the model on the whole dataset, repeating the feature and model selection procedures once more.

Also I would advise against feature selection if the aim is to improve performance (rather than identifying relevant features itself being the goal). Using a regularised model will often perform better.

Answer 3

The thread you mentioned already discusses it in great detail, so I'd skip the parts that were already mentioned there. Answering your question, it depends on what you mean by "one shot feature selection before cv if we do not take response variable into account". For example, if you look at the data and discover that some of the features are very low quality (the data doesn't make sense, it is obviously wrong, or the feature is constant) than yes, you can do this outside of cross-validation. On another hand, doing in-depth exploratory data analysis to pick the features by hand has the same effect as doing similar things algorithmically and there is no reason why "by hand" you won't produce an overfitting model.

Answer 4

In principle, if you want your CV validation scores to reflect what applying an algorithm trained in the manner you did will do on new data, then everything should be part of the cross-validation (or bootstrapping, or whatever else you do along those lines). It's of course the most concerning when we need to make decision on this basis (e.g. Which of several approaches that might be affected differently by deviations from the ideal should I pick?). This ideal is not entirely achievable in practice (except for when you reserve a single separate validation and/or test partition, which is however inefficient for anything but huge datasets) and exploratory data analysis is an obvious example of that.

Some violations of this ideal are worse than others. E.g. you may have to do some EDA to define what your cross-validation scheme should look like (for example, you may discover that you have multiple records per patient and maybe for that reason you should use group-K-fold instead of basic K-fold). Similarly, screening predictors for whether they have zero (or near-zero) variance seems pretty harmless. Creating features solely based on human understanding of the task (e.g. grouping together different mis-spellings of a category name) is usually also totally unproblematic.

Where you are definitely crossing the line into extreme danger is when you do target encoding (representing categories by their mean outcome), that must always be done within the cross-validation loop (or even within an additional CV loop within that). Really, most things that use the prediction target should be considered too dangerous to be done outside the CV loop.

Many other things may be a bit between these extremes. E.g. transforming predictors (e.g. standardization or doing PCA) or imputation of missing predictors ideally belongs in the CV loop, too, but it is less immediately obvious how much (clearly at least a little bit) this would undermine the validity of the CV evaluation. The more you deviate though, the more a final external validation on new data becomes more important (it may of course also be very important for other reasons such as a mismatch of where the training data comes from vs. where the model would be used).