The distribution of the AUC I am wondering how the confidence interval for the Area under the Curve statistic (ROC curves) is derived. I have heard that the AUC can be assumed to be normally distributed, but I am looking for a proof of this statement or a derivation of the confidence intervals
 A: AUC can be viewed as Wilcoxon-Mann-Whitney Test. And here is some demo, where for the R code I posted, I first calculate AUC, then use Wilcoxon-Mann-Whitney Test to calculate the number. Then verify both numbers are the same which is 0.911332. For a hypothesis testing, it is not hard to derive confidence interval. Right? Also I do not remember it Wilcoxon-Mann-Whitney Test requires normal distribution.


A: As hxd1011 said, the AUC is equivalent to the U statistic calculated for the Mann-Whitney-U aka Wilcoxon-rank-sum test, [normalized to [0;1] by the product of observations in each group]. The original paper by Mann and Whitney (1947) contains a proof for approximate normality and the U statistic is approximately Normal distributed even for small samples (~20). In that sense, the AUC could be assumed to be approximately Normal distributed.
To clarify some comments here, note that the test itself does not make any assumption on the distribution of the scores (usually the probability predictions from the model) (i.e. the test is non-parametric).
However, the exact variance of the AUC can only be calculated based on the distribution of the scores (Cortes & Mohri 2005), which typically do not follow a known distribution. There seem to be different suggestions on how to calculate a useful confidence interval, see Cortes & Mohri (2005) for a summary and their (long) formula in Corollary 1 ([4]):
$$ 
\sigma^2(AUC) = \frac{(m+n+1)(m+n)(m+n−1)T((m+n−2)Z_4−(2m−n+3k−10)Z_3)}
{72m^2n^2} + 
\frac{(m+n+1)(m+n)T(m^2−nm+3km−5m+2k^2−nk+12−9k)Z_2}{48m^2n^2} 
− \frac{(m+n+1)^2(m−n)^4 Z_1^2} { 16m^2n^2} 
− \frac{(m+n+1)Q_1 Z_1}{72m^2n^2} +
\frac{kQ_0}{144m^2n^2} 
$$
with:
$$
Z_i = \frac{
\sum^{k−i}_{x=0} 
\left(\begin{smallmatrix}
m+n+1−i \\
x
\end{smallmatrix}\right)
}
{
\sum^k_{x=0}
\left(\begin{smallmatrix}
m+n+1 \\
x
\end{smallmatrix}\right)
}
$$
$$
T = 3((m − n)^2 + m + n) + 2
$$
$$
Q_0 = (m + n + 1)T k^2 + ((−3n^2 + 3mn + 3m + 1)T − 12(3mn + m + n) − 8)k + (−3m^2 +7m + 10n + 3nm + 10)T − 4(3mn + m + n + 1)
$$
$$
Q_1 = T k^3 + 3(m − 1)T k^2 + ((−3n^2 + 3mn − 3m + 8)T − 6(6mn + m + n))k + (−3m^2 +7(m + n) + 3mn)T − 2(6mn + m + n) 
$$
References
Cortes, C., & Mohri, M. (2005). Confidence intervals for the area under the ROC curve. In Advances in neural information processing systems (pp. 305-312). https://cs.nyu.edu/~mohri/pub/area.pdf
