I have a small database (100 samples) with a lot of variance between samples. I am doing repeated cross-validation/holdout, because the results vary notably depending on the splits.

I have 100 features, and I want to do Feature Selection, because the results also vary a lot depending on the features selected.

I haven't found any paper in which feature selection is combined with repeated cross-validation.

What is the correct way of doing it? Any insight will be helpful.


I am doing repeated cross-validation/holdout, because the results vary a LOT depending on the splits.

To me it seems that your model (which you don't talk about but probably should) is unstable.

Instability is a problem when performing K-fold CV or repeated K-fold CV because it leads to high variance and incorrect conclusions. This answer from an influential scientific researcher sheds some light on the intuition behind the issue.

I would therefore argue to first reduce instability in your model, either by choosing a different model which is inherently more stable (e.g. L2 regularized regression or regularized SVM, not a decision tree..) and then do what you wanted to do.

You can definitely perform feature selection via cross validation, but you have to be very careful about the way it is done. In particular to avoid data leakage, and this may involve double (nested) CV

Quoting a Kaggle competitor:

If you use feature selection, and you cross-validate in order to estimate your error, you should include the feature selection step in your CV procedure.

A few threads where this is discussed:


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.