4
$\begingroup$

I'm interested in comparing two AUCs from ROC in two nested models and two AUCs in non-nested models but using the same dataset in a paper. I know DeLong test can be used to compare them, but the formula is kind of complicated for me to do it manually. Also, from the paper, only the mean and the standard error of the AUC are known. Does anyone know how to do it without involving too much calculation or any packages can be used to achieve this?

$\endgroup$
1
  • 1
    $\begingroup$ 0. Welcome to CV.SE. 1. Unfortunately, this isn't enough information to calculate a DeLong (like) test. 2. I give a very rough approximation we can do if we really have to get something out but I would generally recommend against it.. $\endgroup$
    – usεr11852
    Commented Jun 20, 2022 at 2:33

1 Answer 1

5
$\begingroup$

The DeLong test requires the AUC-ROC covariance between the models A and B. If we are willing to accept that the covariance between the two models' AUC-ROC is 0, which is a down-right unreasonable assumption in this case, we can get our $z$-score simply as: $z = \frac{\text{AUC}(\text{Model}_A) - \text{AUC}(\text{Model}_B)}{\sqrt{Var\{\text{AUC}(\text{Model}_A)\} +Var\{\text{AUC}(\text{Model}_B)\} }}$. I would not recommend doing this but hey, it is the best we can hope for given the lack of overall information.

Do note that the covariance would come into the denominator of the calculation shown above, so the (biased) $z$-score shown above is slightly more conservative (i.e. give lower $z$ scores) than what we would expect if the covariance was not $0$. The correct formula would be: $z = \frac{\text{AUC}(\text{Model}_A) - \text{AUC}(\text{Model}_B)}{\sqrt{Var\{\text{AUC}(\text{Model}_A)\} +Var\{\text{AUC}(\text{Model}_B)\} - 2Covar\{ \text{AUC}(\text{Model}_A), \text{AUC}(\text{Model}_B) \} }}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.