I am a little bit confused by the use of feature selection inside a K-fold CV together with hyperparameter tuning.

So I have my dataset. I split in training & test as usual, and work on training set only. Now I would like to do both a feature selection and hyperparameter tuning. Suppose I want to do a recursive feature elimination (RFE).

I start by resampling the 10 folds, in each of which I have an analysis set (for training) and an assessment set (for validation).

I build a reasonable grid of hyperparameters to be evaluated on each fold.

For each point of the grid and each fold I need to do RFE. Now comes the problem: RFE requires to train multiple models (one for each group of features to be evaluated) and evaluate them, to decide the "best" group. But if I use the assessment set to decide the best group of features, how can I use the same set to evaluate the model? Isn't that a bias? Since I am using the group of features that is best "optimized" on the assessment set, shouldn't I use another independent set to evaluate the model?

My feeling is that the unbiased way to do this would be:
For each point of the grid and each fold resample from the analysis set a nested K-fold CV and do choose the RFE "best" group of features on that, than use the best group thus chosen to train the model on the analysis set and evaluate it on the assessment set.

That is:
data $\rightarrow$ training | test
training $\rightarrow$ resample$^1$ | resample$^2$ | ... resample$^{10}$
resample$^i$ = analysis$^i$ | assessment$^i$
analysis$^i$ $\rightarrow$ resample$^i_1$ | resample$^i_2$ | ... resample$^i_K$

Of course there are alternatives: if you have enough data, it is preferable to do the feature selection once and for all on a separate and independent block of data; or you could do feature selection inside each fold but on a small random sampling of the analysis set, without doing an entire nested CV. But here I am wondering whether, theoretically speaking, nested CV is necessary to perform both hyperparameter tuning and feature selection using CV


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.