I have been searching for articles on feature selection and cross-validation, but could not find enough answer to my specific question. I am currently working on an in-house feature selection pipeline for resting-state fMRI data. I use nested CV structure so that I do feature selection in the inner fold. My pipeline is as follows:
for each fold in outer fold for each fold in inner fold run f-test on train data. rank features according to their f-scores. apply k-best algorithm with different *k* values (previously sampled from uniform random distribution) fit model on train data with k-best features, and test on test data. return CV score for each *k* values. ????? end
As you see above, my pipeline basically consists of optimizing k parameter in the k-best feature selection. My question is what to do after feature selection. I have two strategies on that (but I am open for any useful and efficient ideas), which are:
1) Following feature selection, I fit the model on train data in the outer fold with k-best features, which are selected in the inner CV fold.
2) Let's say the first 100 features are found to be the best features with the highest CV score. However, as I rank features by their f-scores, different CV folds may give different features found in the first 100 features. So, to get the best features with a high possibility to have useful information, I take union of features and fit the model on train data in the outer fold with those features.
I am not sure which strategy is better.