Does feature selection and model testing have to be coupled in each fold of the cross-validation?

Question

Quick overview of my data and aims: I have two groups, 50 samples per group, and 6000 features. I want to find the minimal amount of features capable of distinguishing both groups. I know the sample number is not the greatest, but I work with biological samples and getting a total of 100 samples took a lot of work. Besides, I do have additional samples (40 per group) being collected at the moment, and they could be used for further validation and model tuning.

What I think I know: If I perform feature selection (e.g., Boruta) using all my 100 samples and then split them only at the classification stage (e.g., XGBoost with k-fold cross-validation), it would result in data leakage because my test set leaked during feature selection, correct?

Being aware of the above, I was unsure which approach to use: A) start a k-fold cross-validation, perform feature selection, close the folds, take the features that were good across all folds, then "open" another k-fold CV, run the classification algorithm, and assess the results; or B) start the k-fold CV and, in the same fold, perform feature selection plus classification. This way, the selection and classification results are coupled by fold.

Is there any difference between the two methods above? Is one better than the other?

I am asking this for two reasons:

1- I need a panel of important features as soon as possible, but I haven't had time to study classification models enough, so I was hoping I could select the features now and, at a later date, assess the best classification model. 2- I want to test at least five feature selection approaches and ten classification models, so I thought that dividing into two stages would be more organized and better in general. Any thoughts?

Answer 1

Yes, you should run the entire pipeline for each fold

You are right to say that using feature selection on all samples would mean you have data leakage in the following classification cross-validation (CV) step.

You will have the same problem if you separate feature selection (using CV) and classification CV. Just as in the previous scenario, the whole data set is used to inform feature selection (not in the exact same way, but nonetheless).

Think of k-fold CV as doing k times a regular CV. In a regular CV, you run your entire pipeline on the test data, and evaluate on validation data. The exact same thing should be done for each of the folds of k-fold CV to avoid data leakage between the folds.