I recently read this paper: Estimating misclassification error with small samples via bootstrap cross-validation, by Fu et al. (BMC Bioinformatics, 2005).

The authors talk about combining cross validation and bootstrap in order to assess mis-classification error. I was wondering on similar lines, but treating the cross validations as a kind of nested cross validation. For instance, let us assume I have a training data of size 40 samples and 1000 features and another data set that i want to treat as my test data with similar dimensions. I bootstrap my test data and training data in (75/25) proportion, use cross validation to estimate parameters in the training data and then test it on the test set. I was wondering of the problems that such an approach may pose:

  1. The parameters are estimated only on the training bootstrap and may vary from one boot strap to another.
  2. The test error will it be meaningful enough for me to make any conclusion about the model stability since now I am sampling my test data also?
  3. The error due to random sampling may have a larger influence.

I am just learning about the different techniques and this thought crossed my mind. I would like to have your insights on this.

  • $\begingroup$ Bootstrap is usually done by sampling with replacement. You do not have a fixed train/test split (like $75/25$ in your case). The sampling procedure is as follows: if you have $40$ samples, then you randomly select $40$ samples for your train set (this on average yields $63.2\%$ unique instances). The unselected instances become the test set. The number of instances in test set can therefore be different in each iteration. Ad 1) it is normal that results vary from iteration to iteration. This is why you make a lot of repeats and average your results in the end. $\endgroup$ – alesc Mar 23 '15 at 11:24
  • $\begingroup$ @alesc: while you describe bootstrap in the narrower sense, nothing keeps you from drawing another number of cases with replacement. (To me this is like cross validation in the narrower sense: leave-one-out cross validation, and in the wider sense any kind of drawing without replacement) $\endgroup$ – cbeleites Apr 11 '15 at 12:07
  • $\begingroup$ Hi, thanks for your views.. It was recently pointed out to me that the approach in the paper that I cited, often leads to very high bias and should be used only with caution. $\endgroup$ – lekshmi dharmarajan Jul 9 '15 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.