I am comparing few classification models (performance in AUC of ROC curves). I uses 10 times repeated 5 fold cross-validation so I have 50 values of AUC for each model. The key information is, that every 50 resamples are same for each models. So I have same combinations of training set/test for each models.

May I use paired t-test or one sample t-test of AUC differences to test if model performances are different?

Thank you!


In order to use a paired t-test, the assumptions should be met. One of them is that your values should come from a normal distribution. The AUC, however, is a Wilcoxon statistic (see quote below) and therefore violates this assumption.

Now, it just so happens that the Wilcoxon Signed Rank Sum Test tests the same hypothesis as the t-test, but doesn't require normality of the data. I would recommend using that test.

Since I started researching a bit on this, I decided I might as well share it here, as it seems relevant for future visitors: In this paper by Hanley and McNeil from 1982 the distributional characteristics of the AUC are explained:

"It is shown that in such a setting the area represents the probability that a randomly chosen diseased subject is (correctly) rated or ranked with greater suspicion than a randomly chosen non-diseased subject. Moreover, this probability of a correct ranking is the same quantity that is estimated by the already well-studied nonparametric Wilcoxon statistic. "

  1. Calculate the standard errors of AUC scores: $ 𝑆𝐸(π΄π‘ˆπΆ) = \sqrt{\frac{π΄π‘ˆπΆ(1 βˆ’ π΄π‘ˆπΆ) + (𝑁1 βˆ’ 1)(𝑄1 βˆ’ π΄π‘ˆπΆ2) + (𝑁2 βˆ’ 1)(𝑄2 βˆ’ π΄π‘ˆπΆ^2)}{𝑁_1𝑁_2}}$, where $Q1 = \frac{π΄π‘ˆπΆ}{2 βˆ’ π΄π‘ˆπΆ}$ and $Q2= \frac{2AUC^2}{1+AUC}$

  2. Calculate the correlation coefficient $r$ between the two areas. This is probably not zero, because the AUCs "will tend to fluctuate in tandem when derived from a single sample". How to calculate $r$ is outlined in their 1983 paper on page 840.

  3. Calculate the standard error of the difference in AUCs between two models: \begin{equation*} SE\left(AUC_{1}- AUC_{2}\right) = \sqrt{SE^2(AUC_{1}) + SE^2(AUC_{2})-2r\cdot SE(AUC_{1}) SE(AUC_{2})} \end{equation*}.
  4. Calculate $Z$, which is supposed to be a standard normally distributed random variable, so you can just compare it to the good old critical values 1.96: \begin{equation} z = \frac{AUC_1 - AUC_2}{SE(AUC_1 - AUC_2)} \end{equation}

Once you have steps 1 and 2 down, the rest is simple enough.


Your Answer

By clicking β€œPost Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.