I am trying to think of ways of combining bootstrap and cross-validation (CV) to get out-of-sample prediction error and its confidence interval. I was initially thinking of applying this to partial least squares analyses, but the question is more general.
I've read a few papers that seem to do the bootstrap first (i.e. randomly resample the data), then run the model with CV (and repeat B times), to generate a distribution of r^2 values (or whetever metric for prediction error you like). This doesn't make sense to me, as it defeats the point of CV, as during the bootrap i think that makes 36% of your data repeated values.
Alternatively, i was thinking it could be possible to start by calcuting the CV predicted values, then bootstrap-resample these, then calculate the final prediction error stat (e.g. r^2) and repeat to get a distribution of r^2? I'm not sure if this final method is justified however, as normally in bootstrapping you resample the data first and run the model to calculate your output. In this case, the model has been run, and just the predicted values are bootstrap-resampled to generate a distribution.
Is the latter method justified? I've been trying to find papers on this which are written at a relatively accessible level...