Quick overview of my data and aims: I have two groups, 50 samples per group, and 6000 features. I want to find the minimal amount of features capable of distinguishing both groups. I know the sample number is not the greatest, but I work with biological samples and getting a total of 100 samples took a lot of work. Besides, I do have additional samples (40 per group) being collected at the moment, and they could be used for further validation and model tuning.

What I think I know: If I perform feature selection (e.g., Boruta) using all my 100 samples and then split them only at the classification stage (e.g., XGBoost with k-fold cross-validation), it would result in data leakage because my test set leaked during feature selection, correct?

Being aware of the above, I was unsure which approach to use: A) start a k-fold cross-validation, perform feature selection, close the folds, take the features that were good across all folds, then "open" another k-fold CV, run the classification algorithm, and assess the results; or B) start the k-fold CV and, in the same fold, perform feature selection plus classification. This way, the selection and classification results are coupled by fold.

Is there any difference between the two methods above? Is one better than the other?

I am asking this for two reasons:

1- I need a panel of important features as soon as possible, but I haven't had time to study classification models enough, so I was hoping I could select the features now and, at a later date, assess the best classification model. 2- I want to test at least five feature selection approaches and ten classification models, so I thought that dividing into two stages would be more organized and better in general. Any thoughts?


1 Answer 1


Yes, you should run the entire pipeline for each fold

You are right to say that using feature selection on all samples would mean you have data leakage in the following classification cross-validation (CV) step.

You will have the same problem if you separate feature selection (using CV) and classification CV. Just as in the previous scenario, the whole data set is used to inform feature selection (not in the exact same way, but nonetheless).

Think of k-fold CV as doing k times a regular CV. In a regular CV, you run your entire pipeline on the test data, and evaluate on validation data. The exact same thing should be done for each of the folds of k-fold CV to avoid data leakage between the folds.

  • $\begingroup$ Thank you @Scriddie. In my case, where I don't know for sure which machine-learning approach I am going to use, is this appropriate: set the seed, perform the feature selection with k-fold CV, get the list of potential features (genes) and then, when performing the ML training, set the same seed so I can re-do the analysis under the same "randomness"? It would be great to know the panel of interesting features so I can start working on them, even if a few of them end up being discarded when model training. $\endgroup$ Mar 21, 2023 at 15:05
  • $\begingroup$ The feature selection would still be informed by the performance across folds when you choose the model that performs best across folds, so that would not solve the data leakage problem. However, that may not be a deal-breaker. I recommend you set aside about 20% of the data as a test set at the very beginning of your analysis that you never use for anything except to analyze the final model at the end. In this setup, you could run the two steps separately, accept some data leakage in the cross validation, but still be sure that you have an uncontaminated test set. $\endgroup$
    – Scriddie
    Mar 21, 2023 at 15:43
  • 1
    $\begingroup$ Oh, but when I re-do the test, I was planning to follow your advice and use the ML training inside each fold, combined with the feature selection. It is almost like I would be using the set seed to pause the time in between analyses. However, it is probably safer to do what you have just mentioned. I am going to get fresh samples, so I might use most of my dataset now and validate using the new ones. $\endgroup$ Mar 21, 2023 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.