I used bootstrap (N = 1000 samples) to quantificate optimism and subsequently corrected c-index and Somers´ D statistics to select a final model with best predictive accuracy among three "candidate" models. I used then bootstrapping (N = 10 000 samples) on this final model to calculate bootstrapped confidence intervals and p-values of the variables in the final model.

The bootstrap methods, they differ among each other, right? I mean, the first used bootstrap only do validate the model in respect to predictive power and the second makes confidence intervals of variables in final model more realiable?

Thanks for comment


1 Answer 1


By not pre-specifying the full model or pre-specifying the model selection process you are creating a situation where the final bootstrap is incorrect. You will need an outer bootstrap that repeats the optimism bootstrap many times. Or better is to have the inner bootstrap select the model and the outer one compute the optimism. The method you are using will get the confidence intervals wrong (too narrow) by a bit.

  • $\begingroup$ The model selection process was pre-specified in study design. I performed an optimism bootstrap 1000 times and an outer bootstrap (CI) 10 000 times, i.e. 10x more. Did you mean I should performed it at least e.g. 100x or 1000x more? Just a question: CI will be too narrow by a bit as you wrote (a little bit I can accept in respect to the study design) or does it make a really big difference? Thank you! $\endgroup$
    – Juraj
    Commented Nov 10, 2016 at 13:38
  • $\begingroup$ Sounds like you are doing it correctly. A schematic of the algorithm would help. "Too narrow" gets to be more important as the aggressiveness of model selection increases. $\endgroup$ Commented Nov 10, 2016 at 14:44
  • $\begingroup$ @FrankHarrell What's the source of optimistim in vanilla bootstrap? (i.e. why is the estimator optimistically biased)? I went through this in your book on "Regression Modeling Strategies" (Chapter 5), and the source of this optimism wasn't immediately clear (to me). $\endgroup$
    – Josh
    Commented May 26, 2020 at 22:39
  • $\begingroup$ I didn't cover that in the book. More recent work by others shows that with p > N the bootstrap says that the model performance is awful when in truth it is monstrously awful (as revealed by repeated 10-fold CV). $\endgroup$ Commented May 27, 2020 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.