Model comparing with statistical test I am comparing few classification models (performance in AUC of ROC curves). I uses  10 times repeated 5 fold cross-validation so I have 50 values of AUC for each model. The key information is, that every 50 resamples are same for each models. So I have same combinations of training set/test for each models.
May I use paired t-test or one sample t-test of AUC differences to test if model performances are different?
Thank you! 
 A: In order to use a paired t-test, the assumptions should be met. One of them is that your values should come from a normal distribution. The AUC, however, is a Wilcoxon statistic (see quote below) and therefore violates this assumption. 
Now, it just so happens that the Wilcoxon Signed Rank Sum Test tests the same hypothesis as the t-test, but doesn't require normality of the data. I would recommend using that test. 

Since I started researching a bit on this, I decided I might as well share it here, as it seems relevant for future visitors: In this paper by Hanley and McNeil from 1982 the distributional characteristics of the AUC are explained:

"It is shown that in such a setting the area represents the probability that a randomly chosen diseased subject is (correctly) rated or ranked with greater suspicion than a randomly chosen non-diseased subject. Moreover, this probability of a correct ranking is the same quantity that is estimated by the already well-studied nonparametric Wilcoxon statistic. "



*

*Calculate the standard errors of AUC scores: $
() = \sqrt{\frac{(1 − ) + (1 − 1)(1 − 2) + (2 − 1)(2 − ^2)}{_1_2}}$, 
where $Q1 = \frac{}{2 − }$ and $Q2= \frac{2AUC^2}{1+AUC}$

*Calculate the correlation coefficient $r$ between the two areas. This is probably not zero, because the AUCs "will tend to fluctuate in tandem when derived from a single sample". How to calculate $r$ is outlined in their 1983 paper on page 840.

*Calculate the standard error of the difference in AUCs between two models: \begin{equation*}
SE\left(AUC_{1}- AUC_{2}\right) = \sqrt{SE^2(AUC_{1}) + SE^2(AUC_{2})-2r\cdot SE(AUC_{1}) SE(AUC_{2})}
\end{equation*}.

*Calculate $Z$, which is supposed to be a standard normally distributed random variable, so you can just compare it to the good old critical values 1.96: \begin{equation}
z = \frac{AUC_1 - AUC_2}{SE(AUC_1 - AUC_2)}
\end{equation}


Once you have steps 1 and 2 down, the rest is simple enough.  
