I was looking for information about feature selection and crossvalidation, when I found this post:
where there is a discussion about how to use feature selection on cross-validation. I saw that the procedure described is somewhat different from what I use, and I would like to ask if what I do is correct.
What I usually do for feature selection is to perform a search for the best features using the area under roc curve in cross-validation as a function to optimize. I usually perform this search few times, every time with a different cross-validation partitions but with the same number of folds ( 3 to 5) as recomended by Ron kohavi talk ( IJCAI 95). The feature set that appeared as the best one more times is the one I choose. Then, with this feature set I perform 10 fold cross-validation for accuracy prediction. I would appreciate any commentson this procedure. Thanks,