Do you perform cross validation on bootstrap samples? Let's say that I have to chose the best model to predict a variable but my sample size is small. I would like to resample my data using the bootstrap, run each model and evaluate its prediction error with cross validation on each bootstrap sample. In the end I will chose the model with the lowest average prediction error.
Is this a acceptable procedure? If yes is there something I should be aware of while using this procedure?
 A: 
is there something I should be aware of while using this procedure?



*

*You should be aware of the fact that prediction error estimates are subject to bias and variance. With small sample sizes, the variance is the problem.
To get a first idea of the variance of your prediction error estimates, look at the variance of prediction error within and between bootstrapped models that are built using the same input variables. While this can underestimate the true variance, it will give you a first idea of what is going on.
For this, you can e.g. look at the model built using all available input variables, or a model built with input variables that you as an expert for the application think will be a good choice.

*Next, comparing prediction errors is basically a statistical test. Fortunately, you can use paired tests if all competing (surrogate) models were tested using the same test samples. Unfortunately, comparing many models means that you have a massive problem of multiple comparison.
From the results of the variance estimation above, calculate how much better a  model needs to be so you can recognize the improvement. Is that something that you can realistically expect? If not, 


*

*you either need larger sample sizes or 

*you better stick to the first model and report that the available sample size does not allow any further optimization.


*Whenever you do data-driven model optimization$^*$ you need to test the final model with a new independent test data set.
In other words, you need to apply double or nested cross validation (or out-of-bootstrap).
$^*$i.e. tune something [here: which variables to include] by test results.
You may also want to have a look at a similar recent discussion: Grid search on k-fold cross validation
A: The way the bootstrap is applied to estimate prediction error rate is to use the bootstrap samples to estimate the bias of the resubstitution estimate of error rate and adjust it.  Various bootstrap estimators have been devised and shown to work better than the leave-one-out version of cross-validation.  For most population distributions the version called 632+ is the best.  Details about the methods and the literature on it can be found in my book Bootstrap Methods or this book by McLachlan.  This started with Efron 1983 JASA paper "Estimating the error rate of a prediction rule: Improvements on Cross-validation ".
This approach is preferable to the method you have planned.
