Significance testing for comparing ROC areas

Question

I have been analyzing the accuracy of 3 prognostication scores in predicting a certain binary outcome using ROC curves and significance testing for differences in AUCs between the curves (a figure of the ROC curves and the AUCs + 95% confidence intervals for each score is in the post).

As you can see from the figure, Score A has the lowest AUC (0.75). When comparing the AUC for score A against score B, score B's AUC is significantly greater (p=0.02). However, despite score C having a slightly higher AUC and a tighter confidence interval than score B, score C's AUC is not significantly higher than score A's (p=0.08). I was really confused as to why score C's difference would not be significant, given these factors and that all these tests are conducted on the identical dataset (there is no missing data). I am not well versed with the mathematics behind significance testing to compare AUCs and was wondering if there was something that might explain this peculiar trend?

AUC of score A vs. score B: Score B significantly higher (p=0.02)

AUC of score A vs. score C: No difference (p=0.08)

I have found these same results across multiple packages (ex. rocgold() on Stata and the pROC package on R) and different methods (boostrap, delong, etc.) and have gotten the same results across all of these approaches. I am extremely curious why I have been finding these results. Thank you so much!

Answer 1

This is most likely because scores A and B have a higher correlation than A and C.

The DeLong test (1) is a test for two (or more) correlated, or paired, ROC curves. If you look at the paper you will see that it makes a heavy use of the covariance between your two scores. An integral part of the test is to compute the variance-covariance matrix $S$.

When two variables have a high covariance, a small difference in AUC is more likely to be significant than if the covariance was low.

With the pROC package in R you can test this hypothesis by making an unpaired test (with paired = FALSE). Depending on the actual covariance of your data, you should see that both tests are no longer significant.

_{(1) DeLong ER, DeLong DM, Clarke-Pearson DL: Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics 1988,44:837-845. https://www.jstor.org/stable/2531595}