I used bootstrap (N = 1000 samples) to quantificate optimism and subsequently corrected c-index and Somers´ D statistics to select a final model with best predictive accuracy among three "candidate" models. I used then bootstrapping (N = 10 000 samples) on this final model to calculate bootstrapped confidence intervals and p-values of the variables in the final model.

The bootstrap methods, they differ among each other, right? I mean, the first used bootstrap only do validate the model in respect to predictive power and the second makes confidence intervals of variables in final model more realiable?

Thanks for comment


By not pre-specifying the full model or pre-specifying the model selection process you are creating a situation where the final bootstrap is incorrect. You will need an outer bootstrap that repeats the optimism bootstrap many times. Or better is to have the inner bootstrap select the model and the outer one compute the optimism. The method you are using will get the confidence intervals wrong (too narrow) by a bit.

  • $\begingroup$ The model selection process was pre-specified in study design. I performed an optimism bootstrap 1000 times and an outer bootstrap (CI) 10 000 times, i.e. 10x more. Did you mean I should performed it at least e.g. 100x or 1000x more? Just a question: CI will be too narrow by a bit as you wrote (a little bit I can accept in respect to the study design) or does it make a really big difference? Thank you! $\endgroup$ – Juraj Nov 10 '16 at 13:38
  • $\begingroup$ Sounds like you are doing it correctly. A schematic of the algorithm would help. "Too narrow" gets to be more important as the aggressiveness of model selection increases. $\endgroup$ – Frank Harrell Nov 10 '16 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.