I usually have used cross-validation for testing classification performance. However, I read about the article that random sampling (bootstrapping) works better in many cases. I am not sure which one is better in my case.

One of my data have about 300 features and 300 instances - instances are divided into 200 training and 100 test. The class label is binary.

I want to find good features for classification. So I want to test accuracy of classifier. I ran Recursive Feature Elimination (RFE) of python sklearn, so I could get the list of 'feature importance ranking'.

In this case, among 10-fold cross-validation and random sampling,

  1. Use 10-fold cross-validation
  2. (or, random sampling many times)
  3. Calculate mean accuracy of each fold
  4. Reduce least important feature and repeat
  5. The set of features that has highest mean accuray is used as best one.

Which one will be likely to produce better result for classification on test sets from the view of statistics?

  • $\begingroup$ Bootstrapping can be a kind of cross validation. The issue with 10-fold cross-validation is that it can be sensitive to a poor partition of the data. $\endgroup$ – gung - Reinstate Monica Jan 10 '16 at 15:43
  • $\begingroup$ @gung Thank you for comment. There might be no certain answer, but I thought that if I do random sampling many times, more than the number of samples, the result would be more reliable. However, most of papers use only 10-fold cross-validation for test, so I wonder there is a critical point in random sampling. $\endgroup$ – ToBeSpecific Jan 10 '16 at 16:05
  • $\begingroup$ Instead of 10-fold CV you could use k-fold CV which is basically 10-fold CV many times. This would prevent a biased partition of the data as gung put it. But I've heard people advice against repeated CV ( lirias.kuleuven.be/bitstream/123456789/346385/3/… ). Not sure how robust or reliable bootstrapping is. $\endgroup$ – Lennart Jan 10 '16 at 16:57
  • $\begingroup$ @Lennart Thank you for explanation. A paper that bootstrapping 632+ is very reliable (ncbi.nlm.nih.gov/pubmed/14960464), but the other paper says cross-validation is better. Then what about the 'robustness' of feature? In general bootstrap shows better robustness for feature selection, but I am not sure because I have to consider classification performance in my case. $\endgroup$ – ToBeSpecific Jan 10 '16 at 17:20
  • If you use some kind of validation (doesn't matter which) to optimize your model (e.g. by driving the feature reduction), and particularly if you compare many models and/or optimize iteratively, you absolutely need to do a validation of the resulting final model. Whether you do that by a separate validation study, nested cross validation or nested out-of-bootstrap probably won't matter that much.

  • The main difference between the resampling used for cross validation and that for out-of-bootstrap is that bootstrapping resamples with replacement, while cross validation resamples without replacement. In addition, cross validation ensures that within each "run" each sample is tested exactly once.
    I sometimes have questions that are more directly answered by cross repeated/iterated cross validation (stability of predictions), but:

  • We found repeated/iterated k-fold cross validation and out-of-bootstrap resampling having about the same total error based on equal numbers of surrogate models. I'm mostly working with vibrational spectra, 300 features would be quite typical for my data as well; but my features are highly correlated and I usually have far less independent cases (but maybe repeated measurements).
    Here's the paper: Beleites, C.; Baumgartner, R.; Bowman, C.; Somorjai, R.; Steiner, G.; Salzer, R. & Sowa, M. G. Variance reduction in estimating classification error using sparse datasets, Chemom Intell Lab Syst, 79, 91 - 100 (2005).

    Kim, J.-H. Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap , Computational Statistics & Data Analysis , 53, 3735 - 3745 (2009). DOI: 10.1016/j.csda.2009.04.009 reports similar findings.

  • I did not yet thoroughly read the Vanwinckelen paper @Lennart linked above; but at a first glance it looks very promising. Note that while it points out that people may be relying too much on cross validation, it does not compare cross validation vs. bootstrap-based techniques.

  • I also think there's often a deep misunderstanding about what the repetitions/iterations of k-fold cross valiation can do and what they cannot. Importantly, they cannot reduce the variance that is due to the limited number of independent (different) cases tested. What it can and does is: it allows to measure and reduce variance due to model instability. My understanding of the bootstrap-based resampling schemes is that they are similar in that respect.

  • You may want to look into how to choose a cross-validation method questions here as they typically say something about bootstrap as well. Here's a starting point: How to evaluate/select cross validation method?

  • Finally, a totally different thought: at the very least your data driven optimization (feature selection) should use a proper scoring rule, not accuracy. Accuracy is not "well behaved" from a statistical point of view: it has an unnecessarily high variance and on top of that it doesn't necessarily get you the best model.

| cite | improve this answer | |
  • $\begingroup$ Thank you very much for detailed explanation and useful links. It is very helpful advice for me. My method is like 'wrapper method' of feature selection, to find the best subset based on classification score, 10-fold cross-validation without nesting produced unstable output. As you mentioned, it might be due to the limited number of test case. I'd better try nested cross-validation or bootstrap, though I am sure which one is more suitable. Also, thank you for advice for not to use accuracy. I thought it is proper on nearly unbised class, but considering other score like F1 might be better. $\endgroup$ – ToBeSpecific Jan 11 '16 at 5:40
  • $\begingroup$ May I ask one more question if you do not mind? If my purpose is 'stable' solution among various tests(average f1 score of tests), would nested cross-validation be a proper choice? $\endgroup$ – ToBeSpecific Jan 11 '16 at 14:24
  • $\begingroup$ (My guesstimate is that F1 is not a proper scoring rule.) You need to specify what exactly should be stable (stable tested performance of the model is not sufficient to check model stability, sometimes you care only about stability of predictions, sometimes also the model parameters should be stable) - and this will depend on your application. For checking model stability, you need iterated/repeated cross validation. So together with the feature selection: nested iterated/repeated cross validation. $\endgroup$ – cbeleites unhappy with SX Jan 12 '16 at 13:29
  • $\begingroup$ Thank you very much for your answer. What I want to be stable was the performace as you referred. I tried iterated 10-fold cross-validation for test, and it semms to reduce variation. But the difference between results using other features also decreased, so it is also difficult to select best one. (Maybe it is model unstability?) Although I do not know how to handle this proble, I learned that using cross-validation with out considering total model is very dangerous. Thank you again for advice. $\endgroup$ – ToBeSpecific Jan 12 '16 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.