What statistical tests to compare two AUCs from two models on the same dataset?

Let say I build two machine learning classifiers, A and B, on the same dataset.

I obtain the ROC curves for both A and B, and the AUCs value.

What statistical tests should I use to compare these two classifiers. (Let say A is the one I innovate, and B is a baseline model).

Thanks!

• ROC AUC is a Mann-Whitney U statistic, so those confidence intervals are directly relevant here. More discussion in the answers to and comments on this thread: stats.stackexchange.com/questions/189411/…
– Sycorax
May 26 '16 at 3:04
• Thanks @GeneralAbrial . I read the post but I am not quite sure about it. So the Mann-Whitney U statistical test is the way to go? May 26 '16 at 18:23
• The Mann-Whitney U statistic seems like a fairly straightforward statistical hypothesis test: $H_0$ the AUCs equal, $H_1$ they are unequal.
– Sycorax
May 26 '16 at 18:33
• I am not sure if mann-whitney U statistics is the right one to go. May 26 '16 at 21:49

Personally I suggest using a randomized permutation test

Area under curve (AUC) is just one test statistic. You have probably seen that the statistic of A is better than that of B. So it's already established that AUC of A is better than AUC of B. But what is not established is whether this superiority is due to systematic difference, or due to sheer dumb luck.

Therefore, now the question is: is the difference (regardless of which is better than the other) big enough to warrant assuming that the difference is due to systematic differences between methods A and B? In other words:

• What is the probability of you observing that A is better than B under the null hypothesis (which states that A and B have no systematic differences).

Generally, if you go with a randomized permutation test, the procedure to estimate the probability above ($p$ value) is:

1. Calculate AUC of A vs. B (which I assume you already did).
2. Create C_1, such that C_1 is a pair-wisely randomly shuffled list of scores from A and B. In other words, C_1 is a simulation of what a random non-systematic difference looks like.
3. Measure AUC of C_1.
4. Test if AUC of C_1 is better than AUC of A. If yes, increment counter $damn$.
5. Repeat step 2 to 4 $n$ many times, but instead of C_1, use C_i where i $\in \{2, 3, \ldots, n\}$. Usually $n=1000$, but since it's asymptotically consistent, you are free to put larger values of $n$ if you have enough CPU time to go higher.
6. Then, $p = \frac{damn}{n}$.
7. If $p \le \alpha$, then the difference is significant. Usually $\alpha = 0.05$. Else: we don't know (maybe we need larger data).
• Thanks! Several questions. (1) What the proper name for this method? (2) in step 2, what does is mean by "C_1 is a pair-wisely randomly shuffled list of scores from A and B. "? A and B are just a single value respectively. May 26 '16 at 18:24
• (1) Approximate Randomization is what I learned this from. It seems that this is also called a randomized permutation test (as we find C_i that is a random permutation instead of exhaustive). (2) Sorry, some notation abuse here: A and B are method names, but here I used them as arrays. Arrays of what? Arrays of this: since you are doing AUC, you must be calculating the ROC curve, which means that you have the score (sensitivity/specificity) per threshold. So here I assumed that A and B are arrays of such sensitivity/specificity numbers, and C_i is just a randoml pairwise mix between A and B. May 26 '16 at 23:48
• Thanks. Really, can I do that "since you are doing AUC, you must be calculating the ROC curve, which means that you have the score (sensitivity/specificity) per threshold. So here I assumed that A and B are arrays of such sensitivity/specificity numbers, and C_i is just a randoml pairwise mix between A and B." I do have such array (that how the ROC is formed), but the statistical test you mention is good for this kind of comparison? May 27 '16 at 0:48
• Let's wait for gurus and hear their opinion. Feel free to invite them. Basically I've seen Approximate Randomization being used for measuring whether difference in accuracy between A and B is significant. But I havent' seen for AUC. I am not perfectly confident, but at the same time I can't see any reason why wouldn't it work for AUC as the method seems to not be specific to accuracy (as far as I have noticed). Either way let's wait for gurus. May 27 '16 at 1:06
• Cool. Who is Gurus? May 27 '16 at 7:43