# Model comparing with statistical test

I am comparing few classification models (performance in AUC of ROC curves). I uses 10 times repeated 5 fold cross-validation so I have 50 values of AUC for each model. The key information is, that every 50 resamples are same for each models. So I have same combinations of training set/test for each models.

May I use paired t-test or one sample t-test of AUC differences to test if model performances are different?

Thank you!

In order to use a paired t-test, the assumptions should be met. One of them is that your values should come from a normal distribution. The AUC, however, is a Wilcoxon statistic (see quote below) and therefore violates this assumption.

Now, it just so happens that the Wilcoxon Signed Rank Sum Test tests the same hypothesis as the t-test, but doesn't require normality of the data. I would recommend using that test.

Since I started researching a bit on this, I decided I might as well share it here, as it seems relevant for future visitors: In this paper by Hanley and McNeil from 1982 the distributional characteristics of the AUC are explained:

"It is shown that in such a setting the area represents the probability that a randomly chosen diseased subject is (correctly) rated or ranked with greater suspicion than a randomly chosen non-diseased subject. Moreover, this probability of a correct ranking is the same quantity that is estimated by the already well-studied nonparametric Wilcoxon statistic. "

1. Calculate the standard errors of AUC scores: $𝑆𝐸(𝐴𝑈𝐶) = \sqrt{\frac{𝐴𝑈𝐶(1 − 𝐴𝑈𝐶) + (𝑁1 − 1)(𝑄1 − 𝐴𝑈𝐶2) + (𝑁2 − 1)(𝑄2 − 𝐴𝑈𝐶^2)}{𝑁_1𝑁_2}}$, where $Q1 = \frac{𝐴𝑈𝐶}{2 − 𝐴𝑈𝐶}$ and $Q2= \frac{2AUC^2}{1+AUC}$

2. Calculate the correlation coefficient $r$ between the two areas. This is probably not zero, because the AUCs "will tend to fluctuate in tandem when derived from a single sample". How to calculate $r$ is outlined in their 1983 paper on page 840.

3. Calculate the standard error of the difference in AUCs between two models: \begin{equation*} SE\left(AUC_{1}- AUC_{2}\right) = \sqrt{SE^2(AUC_{1}) + SE^2(AUC_{2})-2r\cdot SE(AUC_{1}) SE(AUC_{2})} \end{equation*}.
4. Calculate $Z$, which is supposed to be a standard normally distributed random variable, so you can just compare it to the good old critical values 1.96: $$z = \frac{AUC_1 - AUC_2}{SE(AUC_1 - AUC_2)}$$

Once you have steps 1 and 2 down, the rest is simple enough.