How to do cross-validation when comparing different feature selection methods? I am using SVM for a prediction task. My sample size is small, only N=140. Suppose I want to compare the prediction accuracy when using two different feature selection methods.
Would it be better to:


*

*create a hold-out set that contains e.g. 40 samples, and train the two classifiers (with cross-validation) on the 100 training samples, or

*use cross-validation directly on the 140 samples but just do it twice.
IF I would do (1), may I balance the training and test sets so that they are similar with respect to some variables?
 A: I think your best bet is a paired setup, i.e. working on the exactly same cross-validation or out-of-bootstrap splits. Assuming that your SVM doesn't predict scores, you can then do a McNemar test to compare the results. You may want to run some McNemar test simulation beforehand to check whether the differences you expect to observe have a chance to be significant.
In addition, you should use a sensible number of repetitions/iterations (I'd go for something like 1000 surrogate models) in order to check the stability of the feature selection.
One caveat with the iterations is that they do not contribute new cases over the total number of real cases you have. If your feature selections and the models turn out to be stable (i.e. selector 1 always selects the same features as does selector 2, but the selections may vary between 1 and 2, and the same is true for the SVMs), you can put each single iteration of the cross validation through McNemars, and they should all have practically the same result. 
The correct way of analyzing the test results will be easer for a hold-out but you'd have to pay dearly for that by the much increased uncertainty due to the smaller number of test cases: with 40 test cases, you'd need far more difference between the procedures to claim significance. Frank Harrell will tell you soon that 40 test cases aren't nearly enough to measure the prediction accuracy itself with a useful precision - not to speak of measuring presumably small differences in accurracy.
A: Have you thought about "leave one out cross-validation", the sample is small so you can try this option. This way you don't have to worry about "the balancing" of training and test set.
