5
$\begingroup$

It is me again, asking very similar questions with reference to this post, this time, I am a bit confused whether one should perform feature selection before KFold cross validation. I often times see people do one-shot feature selection prior to modelling, and wonder if this is normal?

In one of the top comment, he mentioned:

I don't think that is (quite) what Hastie, et al. are advocating. The general argument is that if feature selection uses the response then it better be included as part of your CV procedure. If you do predictor screening, e.g., by looking at their sample variances and excluding the predictors with small variation, that is ok as a one-shot procedure.


Wonder if that means we can do one shot feature selection before cv if we do not take response variable into account?

In fact, if I were to do the feature selection within cross validation, then is there any use of doing extensive EDA on the feature selection?


What if there is many multi-collinearity features in my dataset, should I handle it before cross-validation?


Edit: I have been reading up extensively on this issue, but it is really cracking my head because of the uncertainty of when it is "ok" to do feature selection/reduction prior to CV. I chanced upon this very short piece of code from scikit-learn in which they used Ward's Method to remove highly correlated features by choosing a cut off point in the clustering process. I wonder if this is something that I can do outside the loop, and if yes, how does one choose the "threshold"?

$\endgroup$

4 Answers 4

6
$\begingroup$

One of the most challenging elements of this problem is knowing when it's OK to put unsupervised learning steps outside of the CV loop and when they should be fully penalized for by including them inside the loop. Generally speaking, unsupervised learning procedures such as principal components analysis can be unstable, i.e., the loadings of the first principal component will change when computed on a new sample. And unsupervised learning steps to exclude features, such as redundancy analysis and variable clustering, can also be unstable. But their instabilities can either hurt you or help you, i.e., may raise or lower your final $R^2$. So they don't consistently work in your favor. Overfitting in a final predictive discrimination measure such as $R^2$ or pseudo $R^2$ comes from doing things that consistently "work" in your favor, such as doing feature selection using supervised learning, whether manual or automated.

So it's generally OK to keep completely unsupervised learning steps outside the CV loop as pre-processing steps, but sometimes bring them into the loop to make sure your final performance measure doesn't suffer from instability in unsupervised learning.

$\endgroup$
8
  • $\begingroup$ Thanks Professor Frank yet again, I have been reading up extensively on this issue, but it is really cracking my head because of the uncertainty of when it is "ok" to do feature selection/reduction prior to CV. I chanced upon this very short piece of code from scikit-learn in which they used Ward's Method to remove highly correlated features by choosing a cut off point in the clustering process. I wonder if this is something that I can do outside the loop, and if yes, how does one choose the "threshold"? I appended the link in my post. $\endgroup$
    – nan
    Sep 29, 2021 at 12:38
  • 1
    $\begingroup$ In most cases, methods that look only at correlations among predictors are safe to exclude from the resampling loop. But you never know until you try including it in the loop for a few examples. Do note that sometimes the redundancy analysis will exclude an easily predicted feature that is actually the best summary of multiple features and should be the one kept. That's why in general it's not so good to remove features but rather one should combine their effects. This is most readily done by variable clustering + PCA or by a single phase sparse PCA. $\endgroup$ Sep 29, 2021 at 13:09
  • $\begingroup$ Thanks! If I perform single phase sparse PCA, you should still do in the CV loop? $\endgroup$
    – nan
    Sep 29, 2021 at 14:21
  • $\begingroup$ Not usually, but run one simulation with generated data like yours to see if it matters whether it's inside or outside. $\endgroup$ Sep 29, 2021 at 15:50
  • $\begingroup$ This answer seems to suggest that one is justified in performing an unsupervised analysis of features like PCA on the full dataset and then split into train/test sets, but that practice is almost always discouraged. It might have little effect in some cases, and I agree it's likely less of a problem than supervised feature selection, but it's unclear why you shouldn't just do it the "proper" way and never allow the test data to be used at any point in processing the training data. $\endgroup$ Sep 29, 2021 at 16:44
8
$\begingroup$

Cross-validation is a means of estimating the performance of a method for fitting a model, rather than of the model itself, so all steps in fitting the model (including feature selection and optimising the hyper-parameters) need to be performed independently in each fold of the cross-validation procedure. If you don't do this, then you will end up with an optimistically biased performance estimate. See my (with Mrs Marsupial) paper on this topic

GC Cawley and NLC Talbot, "On over-fitting in model selection and subsequent selection bias in performance evaluation", The Journal of Machine Learning Research 11, 2079-2107 (pdf)

I tend to use nested cross-validation to get an unbiased performance estimate, but if you don't need an unbiased performance estimate, just choose between competing methods (that don't have too many degrees of freedom, i.e. not feature selection!) then that often isn't necessary in practice, see Wainer and Cawley

J Wainer and G Cawley, "Nested cross-validation when selecting classifiers is overzealous for most practical applications", Expert Systems with Applications 182, 115 (doi:10.1016/j.eswa.2021.115222)

Once you have that performance estimate, then retrain the model on the whole dataset, repeating the feature and model selection procedures once more.

Also I would advise against feature selection if the aim is to improve performance (rather than identifying relevant features itself being the goal). Using a regularised model will often perform better.

$\endgroup$
6
  • $\begingroup$ Thanks for this, I have updated my question! Just to be clear, when you say retrain the model on the whole dataset, we use the best hyperparameter in the feature selection as well right $\endgroup$
    – nan
    Sep 29, 2021 at 6:50
  • $\begingroup$ If you have multicolinearity, regularisation is probably going to be a better solution than feature selection from a generalisation perspective. Even Millar in his monograph on feature subset selection makes that recommendation (in the appendix) and it matches my experience (I helped organise a challenge on causal and non-causal feature selection - not performing any feature selection tended to work just as well, IIRC!) $\endgroup$ Sep 29, 2021 at 6:53
  • $\begingroup$ Generally feature selection is challenging - you have one binary degree of freedom for each feature (which is often a lot) and very little in the way of theoretical underpinnings. With regularisation you tend to have one continuous degree of freedom, which makes it more difficult to over-fit the selection criterion. $\endgroup$ Sep 29, 2021 at 7:01
  • 1
    $\begingroup$ Indeed, I have been kind of stuck on how to choose features. I have 30 features, and am very tempted to remove some because some of them are more or less highly correlated with the other (i.e. tumor's radius, and tumor's perimeter, are related by the circumference formula to a large extent - though not perfect) $\endgroup$
    – nan
    Sep 29, 2021 at 7:36
  • 1
    $\begingroup$ +1 for Mrs. Marsupial $\endgroup$
    – bdeonovic
    Sep 29, 2021 at 13:52
5
$\begingroup$

The thread you mentioned already discusses it in great detail, so I'd skip the parts that were already mentioned there. Answering your question, it depends on what you mean by "one shot feature selection before cv if we do not take response variable into account". For example, if you look at the data and discover that some of the features are very low quality (the data doesn't make sense, it is obviously wrong, or the feature is constant) than yes, you can do this outside of cross-validation. On another hand, doing in-depth exploratory data analysis to pick the features by hand has the same effect as doing similar things algorithmically and there is no reason why "by hand" you won't produce an overfitting model.

$\endgroup$
6
  • $\begingroup$ Thanks for this, I have updated my question to ask whether multi-collinearity features should be handled before cross-validation? $\endgroup$
    – nan
    Sep 29, 2021 at 6:50
  • $\begingroup$ @nan you mean perfect multicollinearity or just correlation between features? Also, it's something you need to bother about it when using linear regression, but not in case of most of the machine learning models. $\endgroup$
    – Tim
    Sep 29, 2021 at 6:55
  • 3
    $\begingroup$ +1 Performing the feature selection manually is probably worse as the "researcher degrees of freedom" are difficult to account for in the performance evaluation (and in describing exactly what you did, so it is difficult to make the work reproducible). We ought to avoid "CyborgML" (AutoML where the researcher becomes part of the mechanism) where possible ;o) I always dread to see "we determined the hyper-parameters via a preliminary exploration" (or similar) when I am reviewing a paper, as is means the results are likely to be questionable (due to CyborgML overfitting). $\endgroup$ Sep 29, 2021 at 6:55
  • $\begingroup$ Dear both, thanks for the insights. There are a few features with 0.9 and above in correlation. So I am stuck in between whether I should justify why I need to remove those correlated features, or if there is any other way to help me choose the features I want. Is there any good methods to help me select the features? I have seen people using a tree to fit the model and get the feature_importance_ from them, and selecting the top N features, and retrain it again, is this recommended? $\endgroup$
    – nan
    Sep 29, 2021 at 7:00
  • 4
    $\begingroup$ @nan in general you do not need to remove any features. In machine learning, the models can deal with "selecting" the features they use by themselves. You can use regularization to facilitate it even more. You remove the features if you have really good reasons for that (e.g. the former didn't work). On another hand, if using linear regression for inference, you need to bother about perfect multicollinearity, and the features you select should be based on your research question, not exploratory data analysis and cherry-picking. $\endgroup$
    – Tim
    Sep 29, 2021 at 7:07
3
$\begingroup$

In principle, if you want your CV validation scores to reflect what applying an algorithm trained in the manner you did will do on new data, then everything should be part of the cross-validation (or bootstrapping, or whatever else you do along those lines). It's of course the most concerning when we need to make decision on this basis (e.g. Which of several approaches that might be affected differently by deviations from the ideal should I pick?). This ideal is not entirely achievable in practice (except for when you reserve a single separate validation and/or test partition, which is however inefficient for anything but huge datasets) and exploratory data analysis is an obvious example of that.

Some violations of this ideal are worse than others. E.g. you may have to do some EDA to define what your cross-validation scheme should look like (for example, you may discover that you have multiple records per patient and maybe for that reason you should use group-K-fold instead of basic K-fold). Similarly, screening predictors for whether they have zero (or near-zero) variance seems pretty harmless. Creating features solely based on human understanding of the task (e.g. grouping together different mis-spellings of a category name) is usually also totally unproblematic.

Where you are definitely crossing the line into extreme danger is when you do target encoding (representing categories by their mean outcome), that must always be done within the cross-validation loop (or even within an additional CV loop within that). Really, most things that use the prediction target should be considered too dangerous to be done outside the CV loop.

Many other things may be a bit between these extremes. E.g. transforming predictors (e.g. standardization or doing PCA) or imputation of missing predictors ideally belongs in the CV loop, too, but it is less immediately obvious how much (clearly at least a little bit) this would undermine the validity of the CV evaluation. The more you deviate though, the more a final external validation on new data becomes more important (it may of course also be very important for other reasons such as a mismatch of where the training data comes from vs. where the model would be used).

$\endgroup$
2
  • $\begingroup$ Thanks, if I understood your idea correctly, you are saying that EDA can still be done, because it is harmless and can uncover insights. However, ideally, things like standardization/pca (or even feature selection) should be in the inner loop of cv $\endgroup$
    – nan
    Sep 29, 2021 at 7:48
  • 2
    $\begingroup$ Yes. Not all EDA is harmless though. To illustrate a problematic use of EDA: E.g. imagine we look at outcomes by categories during our EDA and then decide to pool some categories based on that. We need to watch ourselves so we don't carelessly do things like. Of course, we may get ideas like that during EDA, but then after noticing that we have a lot of categories during EDA (and possibly even the idea that some might have similar responses), we should really then to try different way / thresholds for combining categories via CV (rather than fixing the criteria manually based on EDA). $\endgroup$
    – Björn
    Sep 29, 2021 at 9:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.