I have been reading some of the posts about feature selection and cross-validation but I still have questions about the correct procedure.
Suppose I have a dataset with 10 features and I want to select the best features. Also suppose I am using one-nearest neighbor classifier. Can I perform an exhaustive search using cross-validation to estimate the error rate as guide to choose the best features? Something like the following pseudo code
for i=1:( 2^10 -1)
error(i)= crossval(1-nn, selected_fetures(i))
end
i=find(erro(i)==min(error(i));
selected_fetures= selected_features(i);
What I'm trying to explain in this pseudo code is that I'm running the cross validation for all possible combinations of features and choose the combination that gives the minimum error.
I think that this procedure is correct because I am performing an exhaustive search. The choice of the features was not based on the entire dataset, but on the average error on each partition. Am I overfitting the model with such feature selection?