My dataset is not quite large enough to be split into training and testing sets, so I am using bootstrapped optimism to account for overfitting when reporting model performance.

The cases in my data set have a low prevalence, such that in some iterations of sampling they're missed. To account for this I've increased the number of times I am sampling and consider set the optimism, for those instances that didn't sample enough (or any) cases for the model to converge, as NA. I wanted to get a sense for whether that made sense, or if anyone has any experience doing a stratified sampling approach for bootstrapped optimism.

I am using Frank Harrell's 1996 stats medicine paper (Tutorial in biostatistics: Multivariable Prognostic models..") as a guide. I know that RMS has the validate function that can do this for lm and cph models, but I am using neither of these.


1 Answer 1


I can't comment yet ('low' reputation) so I'm posting this as an answer.

Increasing the number of bootstrap replications will only decrease the standard error of your bootstrap statistic. It may mitigate the problem you mentioned but it won't solve it.

I'm at a very similar situation: small sample (<100) with a ratio of cases/controls of about 1:4. I've found that balanced bootstrap and stratified cross-validation give more stable results. I found empirically that if we don't account for low prevalence of cases it wasn't that uncommon to get a sample without any cases.

For balanced bootstrap you sample the cases from the cases empirical distribution, doing the same for the controls. This way the case prevalence in the bootstrap replications will be fixed and equal to the prevalence in the whole empirical distribution. The same logic applied to CV, though the procedure is called 'stratified' rather than 'balanced'.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.