1
$\begingroup$

I am trying to estimate confidence bounds for prediction metrics of a binary-outcome logistic regression model. In particular, the AUC and the Brier score.

I've looked at many other posts on this website, but each seems to have differing and sometimes conflicting ways of computing CIs using either bootstrapping or cross validation.

I decided to use the rms package and the validate method to estimate these metrics. The validate method uses optimism-corrected bootstrap resampling to compute metrics. However, it does not provide confidence intervals.

A comment on a post on another website by the author of the package indicates the following method of obtaining confidence bounds:

Instantiate metric vector v
Loop 1...n:
   Perform bootstrap resampling of original data d, store in d'
   Compute optimism-corrected bootstrap metrics of model w.r. to d' using B bootstrap samples
   Accumulate in v

Compute CIs using quantiles on v

However, due to the larger size of my dataset (~350K samples), this method of calculating CIs takes a long time to run. I'm trying to replace parts of the bootstrapping process with cross-validation.

Would it be valid to replace the computation of optimism-corrected bootstrap metrics with CV as follows?

Instantiate metric vector v
Loop 1...n:
   Perform bootstrap resampling of original data d, store in d'
   Compute averaged K-fold CV metrics of model w.r. to d'
   Accumulate in v

Compute CIs using quantiles on v

Or further, is it also appropriate to replace the outer most bootstrap sampling with CV as well?

Instantiate metric vector v
Loop 1...n:
   Compute averaged K-fold CV metrics of model w.r. to d
   Accumulate in v

Compute CIs using quantiles on v

I've seen other posts such as this showing how to compute confidence intervals and a t-test statistic. However, I wasn't sure if this would work for bounded metrics such as AUC and the Brier score.

I've also seen another post that explains that recall metrics can't be just averaged across CV folds which lays some doubts about the Compute averaged K-fold CV metrics ... step. Meanwhile for yet another post I'm unable to figure out what the commenter meant by bootstrapping of the resampled mean and the corresponding paragraph in their paper.

Appreciate any help!

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.