Is this a correct procedure for feature selection using cross-validation?

Question

I was looking for information about feature selection and crossvalidation, when I found this post:

Feature selection for "final" model when performing cross-validation in machine learning.

where there is a discussion about how to use feature selection on cross-validation. I saw that the procedure described is somewhat different from what I use, and I would like to ask if what I do is correct.

What I usually do for feature selection is to perform a search for the best features using the area under roc curve in cross-validation as a function to optimize. I usually perform this search few times, every time with a different cross-validation partitions but with the same number of folds ( 3 to 5) as recomended by Ron kohavi talk ( IJCAI 95). The feature set that appeared as the best one more times is the one I choose. Then, with this feature set I perform 10 fold cross-validation for accuracy prediction. I would appreciate any commentson this procedure. Thanks,

Jorge

Answer 1

I think there is still a potential for bias here unless the test set used in your 10 fold CV is separate. I think there is a chance that the test set in each fold of your cross-validation (CV) may have already been used in choosing the feature set. This may mean that your model will not generalize as well on genuinely unseen data.

However, I think this approach is definitely less problematic then performing feature selection once prior to CV. A colleague of mine compared this approach (on a small datase, N=40) to one where feature selection is carried out within each fold of the CV and found it did not make much difference to the performance estimate which suggests that the performance estimate was not heavily biased.