I usually have used cross-validation for testing classification performance. However, I read about the article that random sampling (bootstrapping) works better in many cases. I am not sure which one is better in my case.
One of my data have about 300 features and 300 instances - instances are divided into 200 training and 100 test. The class label is binary.
I want to find good features for classification. So I want to test accuracy of classifier. I ran Recursive Feature Elimination (RFE) of python sklearn, so I could get the list of 'feature importance ranking'.
In this case, among 10-fold cross-validation and random sampling,
- Use 10-fold cross-validation
- (or, random sampling many times)
- Calculate mean accuracy of each fold
- Reduce least important feature and repeat
- The set of features that has highest mean accuray is used as best one.
Which one will be likely to produce better result for classification on test sets from the view of statistics?