0
$\begingroup$

Confidence intervals around two AUCs were generated using bootstrapping with a confidence level of 95 %. The CIs overlap each other, hence I conduct significance testing by bootstrapping the difference of AUC CIs. This confidence interval did not include zero, and therefore I conclude that there is an effect when using a 5 % significance level. Is this the correct approach? If not, how am I supposed to conduct the significance testing?

$\endgroup$
1
$\begingroup$

95% confidence intervals can still overlap and the difference be significantly different. It is better to think about the intervals as less of straight lines (like they are usually thought of) and instead as curves (Geoff Cumming calls them "cat eyes"):

enter image description here

The point here is that, even if two confidence intervals overlap, the overlapping bits are so far in the tails of both distributions that for a point to land in that little region where they overlap is unlikely, given that the null hypothesis is true.

So what you did appears to be fine with me: Get the difference between the two and bootstrap the difference to see if that 95% confidence interval for the difference includes zero.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.