I am working with a feature selection problem. What I am trying to do is find optimal subset of features for classification.
My data consist of 100 features and 300 instances, and class label is binary. My aim is to detect best feature set that is most correlated to the class label.
First I applied Recursive Feature Elimination(RFE) by Guyon et al. It calculate the weight vectors of all features using SVM, and remove one feature of least weight. It repeats until final subset containing only 1 feature is tested. In this way, all ranking of feature can be acquired.
But it does not suggest guideline for how many features should be selected, at least as I understood.
To find the optimal(Though it is heuristic) subset, I first tried 10-fold cross-validation test and calcuated mean f1 score, removing features one by one backward along the ranking derived from RFE. Namely, 10-fold cv with 100 features, 99 features, ... 1 feature.
However, the result is not uniform. Sometimes 10-feature subset is best, and other times 50-feature subset is best. I think it is due to the lack of test cases, thus I am thinking of testing with random sampling or bagging or repeated cross-validation or something.
This is why I asked previous question about difference between cross-validation and sampling.
In this case, what would be the most appropriate model for the test? My goal is to get stable result(feature set). Also, is there any recommended classifier for this kind of work?