Overlapping Confidence Intervals and Statistical Significance - using bootstrapping

Question

Confidence intervals around two AUCs were generated using bootstrapping with a confidence level of 95 %. The CIs overlap each other, hence I conduct significance testing by bootstrapping the difference of AUC CIs. This confidence interval did not include zero, and therefore I conclude that there is an effect when using a 5 % significance level. Is this the correct approach? If not, how am I supposed to conduct the significance testing?

Answer 1

95% confidence intervals can still overlap and the difference be significantly different. It is better to think about the intervals as less of straight lines (like they are usually thought of) and instead as curves (Geoff Cumming calls them "cat eyes"):

The point here is that, even if two confidence intervals overlap, the overlapping bits are so far in the tails of both distributions that for a point to land in that little region where they overlap is unlikely, given that the null hypothesis is true.

So what you did appears to be fine with me: Get the difference between the two and bootstrap the difference to see if that 95% confidence interval for the difference includes zero.