4
$\begingroup$

I was looking for information about feature selection and crossvalidation, when I found this post:

Feature selection for "final" model when performing cross-validation in machine learning.

where there is a discussion about how to use feature selection on cross-validation. I saw that the procedure described is somewhat different from what I use, and I would like to ask if what I do is correct.

What I usually do for feature selection is to perform a search for the best features using the area under roc curve in cross-validation as a function to optimize. I usually perform this search few times, every time with a different cross-validation partitions but with the same number of folds ( 3 to 5) as recomended by Ron kohavi talk ( IJCAI 95). The feature set that appeared as the best one more times is the one I choose. Then, with this feature set I perform 10 fold cross-validation for accuracy prediction. I would appreciate any commentson this procedure. Thanks,

Jorge

$\endgroup$
  • $\begingroup$ Welcome to the site again, and thanks for posting a separate question, as suggested! $\endgroup$ – chl Mar 26 '12 at 17:30
1
$\begingroup$

I think there is still a potential for bias here unless the test set used in your 10 fold CV is separate. I think there is a chance that the test set in each fold of your cross-validation (CV) may have already been used in choosing the feature set. This may mean that your model will not generalize as well on genuinely unseen data.

However, I think this approach is definitely less problematic then performing feature selection once prior to CV. A colleague of mine compared this approach (on a small datase, N=40) to one where feature selection is carried out within each fold of the CV and found it did not make much difference to the performance estimate which suggests that the performance estimate was not heavily biased.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.