I recently read this paper: Estimating misclassification error with small samples via bootstrap cross-validation, by Fu et al. (BMC Bioinformatics, 2005).
The authors talk about combining cross validation and bootstrap in order to assess mis-classification error. I was wondering on similar lines, but treating the cross validations as a kind of nested cross validation. For instance, let us assume I have a training data of size 40 samples and 1000 features and another data set that i want to treat as my test data with similar dimensions. I bootstrap my test data and training data in (75/25) proportion, use cross validation to estimate parameters in the training data and then test it on the test set. I was wondering of the problems that such an approach may pose:
- The parameters are estimated only on the training bootstrap and may vary from one boot strap to another.
- The test error will it be meaningful enough for me to make any conclusion about the model stability since now I am sampling my test data also?
- The error due to random sampling may have a larger influence.
I am just learning about the different techniques and this thought crossed my mind. I would like to have your insights on this.