# Efron bootstrapped confidence intervals does not include corrected value of metric

So in order to correct for optimism on my model I have used the Efron bootstrapping to calculate the optimism to correct my apparent value of several validation metrics for my model (repeated 500 times).

Now, fx in one case my apparent AUC value is 0.70, and the corrected value is 0.65. However, the calculated confidence intervals (also based on the Efron bootstrapping) is [0.66, 0.81]. So basically not containing the corrected value. And this holds for several metrics. Intuitively (at least for me) that seems weird, or at least it maybe means that my model is really bad (which may the case) ? Or maybe it should be compared to the apparent value ?

Anyways, is this normal behavior, or am I doing something wrong ?

For clarity, the way I calculate the CI is by taking the distribution of the bootstrapped metric values (i.e. model developed on the bootstrap sample and tested on the bootstrap sample (not the original bootstrap sample)), sorting from low to high (for the low CI), and finding the (0.025*500)th index/value, and then the opposite for the high CI.

• @DenverDang I haven't used that solution but see no reason why it shouldn't work. Try small-scale tests to ensure that what you get agrees with what you get with boot.ci() in the R boot package. I have rolled my own boot objects to do this, in a context where I had to remove some values to deal with errors. See the manual for boot() and the code for the boot:::boot.return() function to see how to generate them. Or adapt the code used in the underlying functions boot:::bca.ci(), boot:::empinf(), and boot:::empinf.reg() functions.