Efron bootstrapped confidence intervals does not include corrected value of metric So in order to correct for optimism on my model I have used the Efron bootstrapping to calculate the optimism to correct my apparent value of several validation metrics for my model (repeated 500 times).
Now, fx in one case my apparent AUC value is 0.70, and the corrected value is 0.65. However, the calculated confidence intervals (also based on the Efron bootstrapping) is [0.66, 0.81]. So basically not containing the corrected value. And this holds for several metrics. Intuitively (at least for me) that seems weird, or at least it maybe means that my model is really bad (which may the case) ? Or maybe it should be compared to the apparent value ?
Anyways, is this normal behavior, or am I doing something wrong ?
For clarity, the way I calculate the CI is by taking the distribution of the bootstrapped metric values (i.e. model developed on the bootstrap sample and tested on the bootstrap sample (not the original bootstrap sample)), sorting from low to high (for the low CI), and finding the (0.025*500)th index/value, and then the opposite for the high CI.
 A: This behavior can occur when the sample-based estimate of the parameter is biased from the population value. You documented such bias in your bootstrapping. Insofar as bootstrap sampling from your original data represents sampling of the original data from the population, you showed that your AUC estimate from sampling is biased downward from that in the population from which you sampled by about 0.05 units. This page shows a similar result for confidence intervals of the necessarily biased plug-in estimate of Shannon entropy, and this page goes into what I remember as excruciating detail about different ways to estimate confidence intervals and how the way that you calculated confidence intervals can lead to this behavior when there is bias.
Efron and other realized this problem from early on, and additional problems arising from skew in the distributions of estimates. The BCa bootstrap, which attempts to correct for both bias and skew, is implemented in standard statistical software.
