I've seen taught several places that feature selection needs to be inside the CV training loop. Here are three examples where I have seen this:

Feature selection and cross-validation

Nested cross-validation and feature selection: when to perform the feature selection?


...you must include feature selection within the inner-loop when you are using accuracy estimation methods such as cross-validation. This means that feature selection is performed on the prepared fold right before the model is trained. A mistake would be to perform feature selection first to prepare your data, then perform model selection and training on the selected features...

Here is an example from the sklearn docs, that shows how to do recursive feature elimination with regular n-fold cross validation.

However I'd like to do recursive feature elimination inside random/grid CV, so that "feature selection is performed on the prepared fold right before the model is trained (on the random/grid selected params for that fold)", so that data from other folds influence neither feature selection nor hyperparameter optimization.

Is this possible natively with sklearn methods and/or pipelines? Basically, I'm trying to find an sklearn native way to do this before I go code it from scratch.


If you want to search over the number of features to retain, then you need some sort of cross-validation, and since as you point out this needs to be done inside the training set of the main model fit, this will require nested cross-validation. If that's not a computational problem for you, then sklearn makes this pretty simple.

pipe = Pipeline(steps=[('feat_sel', RFECV(...)),
                       # ...other preprocessing?...
                       ('model', LogisticRegression(...)) # or whatever
search = RandomizedSearchCV(estimator=pipe, ...)
search.fit(X, y)

Note that you could provide your own cross-validation instead of the default k-fold, if you need to save some computation time.

The other option is to use RFE instead of RFECV, which requires a fixed number of features to use, but then doesn't need its own cross-validation; this seems best if you have some domain or previous model information to suggest the right number of features in advance (be sure not to use your whole training set to determine that!).

  • $\begingroup$ Thank you @Ben Rein, that's the answer! $\endgroup$ – 3z33etm May 7 '20 at 14:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.