So in order to correct for optimism on my model I have used the Efron bootstrapping to calculate the optimism to correct my apparent value of several validation metrics for my model (repeated 500 times).
Now, fx in one case my apparent AUC value is 0.70, and the corrected value is 0.65. However, the calculated confidence intervals (also based on the Efron bootstrapping) is [0.66, 0.81]. So basically not containing the corrected value. And this holds for several metrics. Intuitively (at least for me) that seems weird, or at least it maybe means that my model is really bad (which may the case) ? Or maybe it should be compared to the apparent value ?
Anyways, is this normal behavior, or am I doing something wrong ?
For clarity, the way I calculate the CI is by taking the distribution of the bootstrapped metric values (i.e. model developed on the bootstrap sample and tested on the bootstrap sample (not the original bootstrap sample)), sorting from low to high (for the low CI), and finding the (0.025*500)th index/value, and then the opposite for the high CI.