10
$\begingroup$

Let say I build two machine learning classifiers, A and B, on the same dataset.

I obtain the ROC curves for both A and B, and the AUCs value.

What statistical tests should I use to compare these two classifiers. (Let say A is the one I innovate, and B is a baseline model).

Thanks!

$\endgroup$
4
  • 3
    $\begingroup$ ROC AUC is a Mann-Whitney U statistic, so those confidence intervals are directly relevant here. More discussion in the answers to and comments on this thread: stats.stackexchange.com/questions/189411/… $\endgroup$
    – Sycorax
    May 26 '16 at 3:04
  • $\begingroup$ Thanks @GeneralAbrial . I read the post but I am not quite sure about it. So the Mann-Whitney U statistical test is the way to go? $\endgroup$ May 26 '16 at 18:23
  • $\begingroup$ The Mann-Whitney U statistic seems like a fairly straightforward statistical hypothesis test: $H_0$ the AUCs equal, $H_1$ they are unequal. $\endgroup$
    – Sycorax
    May 26 '16 at 18:33
  • $\begingroup$ I am not sure if mann-whitney U statistics is the right one to go. $\endgroup$ May 26 '16 at 21:49
1
$\begingroup$

Personally I suggest using a randomized permutation test

Area under curve (AUC) is just one test statistic. You have probably seen that the statistic of A is better than that of B. So it's already established that AUC of A is better than AUC of B. But what is not established is whether this superiority is due to systematic difference, or due to sheer dumb luck.

Therefore, now the question is: is the difference (regardless of which is better than the other) big enough to warrant assuming that the difference is due to systematic differences between methods A and B? In other words:

  • What is the probability of you observing that A is better than B under the null hypothesis (which states that A and B have no systematic differences).

Generally, if you go with a randomized permutation test, the procedure to estimate the probability above ($p$ value) is:

  1. Calculate AUC of A vs. B (which I assume you already did).
  2. Create C_1, such that C_1 is a pair-wisely randomly shuffled list of scores from A and B. In other words, C_1 is a simulation of what a random non-systematic difference looks like.
  3. Measure AUC of C_1.
  4. Test if AUC of C_1 is better than AUC of A. If yes, increment counter $damn$.
  5. Repeat step 2 to 4 $n$ many times, but instead of C_1, use C_i where i $\in \{2, 3, \ldots, n\}$. Usually $n=1000$, but since it's asymptotically consistent, you are free to put larger values of $n$ if you have enough CPU time to go higher.
  6. Then, $p = \frac{damn}{n}$.
  7. If $p \le \alpha$, then the difference is significant. Usually $\alpha = 0.05$. Else: we don't know (maybe we need larger data).
$\endgroup$
7
  • $\begingroup$ Thanks! Several questions. (1) What the proper name for this method? (2) in step 2, what does is mean by "C_1 is a pair-wisely randomly shuffled list of scores from A and B. "? A and B are just a single value respectively. $\endgroup$ May 26 '16 at 18:24
  • $\begingroup$ (1) Approximate Randomization is what I learned this from. It seems that this is also called a randomized permutation test (as we find C_i that is a random permutation instead of exhaustive). (2) Sorry, some notation abuse here: A and B are method names, but here I used them as arrays. Arrays of what? Arrays of this: since you are doing AUC, you must be calculating the ROC curve, which means that you have the score (sensitivity/specificity) per threshold. So here I assumed that A and B are arrays of such sensitivity/specificity numbers, and C_i is just a randoml pairwise mix between A and B. $\endgroup$
    – caveman
    May 26 '16 at 23:48
  • 1
    $\begingroup$ Thanks. Really, can I do that "since you are doing AUC, you must be calculating the ROC curve, which means that you have the score (sensitivity/specificity) per threshold. So here I assumed that A and B are arrays of such sensitivity/specificity numbers, and C_i is just a randoml pairwise mix between A and B." I do have such array (that how the ROC is formed), but the statistical test you mention is good for this kind of comparison? $\endgroup$ May 27 '16 at 0:48
  • $\begingroup$ Let's wait for gurus and hear their opinion. Feel free to invite them. Basically I've seen Approximate Randomization being used for measuring whether difference in accuracy between A and B is significant. But I havent' seen for AUC. I am not perfectly confident, but at the same time I can't see any reason why wouldn't it work for AUC as the method seems to not be specific to accuracy (as far as I have noticed). Either way let's wait for gurus. $\endgroup$
    – caveman
    May 27 '16 at 1:06
  • 1
    $\begingroup$ Cool. Who is Gurus? $\endgroup$ May 27 '16 at 7:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.