I am wondering how the confidence interval for the Area under the Curve statistic (ROC curves) is derived. I have heard that the AUC can be assumed to be normally distributed, but I am looking for a proof of this statement or a derivation of the confidence intervals

  • 1
    $\begingroup$ Can you give a reference to the claim about normality of the AUC? $\endgroup$ – Antoine Jun 1 '16 at 11:37
  • $\begingroup$ @Antoine I've actually heard this statement from a colleague of mine, I haven't been able to find any references about this. I am currently looking at the 'binormal' derivation of the ROC curve - are you familiar with pther ways to estimate the confidence intervals for the AUC ? $\endgroup$ – WeakLearner Jun 1 '16 at 11:44
  • $\begingroup$ No, I am not. I only think that your question is quite important. $\endgroup$ – Antoine Jun 1 '16 at 12:17
  • $\begingroup$ @Antoine just found this: ncss.com/wp-content/themes/ncss/pdf/Procedures/NCSS/…, they refer to a transformation of the AUC which is 'closer' to normality, I am trying to find the text they refer to $\endgroup$ – WeakLearner Jun 1 '16 at 12:23
  • 2
    $\begingroup$ AUC can't have a normal distribution because the normal distribution has positive probability over the whole real line. AUC is bounded between 0 and 1. $\endgroup$ – Sycorax Jun 1 '16 at 14:55

AUC can be viewed as Wilcoxon-Mann-Whitney Test. And here is some demo, where for the R code I posted, I first calculate AUC, then use Wilcoxon-Mann-Whitney Test to calculate the number. Then verify both numbers are the same which is 0.911332. For a hypothesis testing, it is not hard to derive confidence interval. Right? Also I do not remember it Wilcoxon-Mann-Whitney Test requires normal distribution.

enter image description here enter image description here

  • $\begingroup$ The Wilcoxon Mann Whitney test is a non-parametric test: it makes no assumptions about the underlying distribution of the data. $\endgroup$ – Sycorax Jun 1 '16 at 14:55
  • $\begingroup$ @hxd1011 Given the Wilcoxon Mann Whitney test is non-parametric as asserted by Sycorax, does it then follow that the distribution of the AUC is non-parametric? Specifically, if there is no parametric distribution for AUC, are confidence intervals around the AUC best computed from the empirical distribution (e.g. using a bootstrap)? $\endgroup$ – Greenstick Jul 25 '18 at 23:26

As hxd1011 said, the AUC is equivalent to the U statistic calculated for the Mann-Whitney-U aka Wilcoxon-rank-sum test, [normalized to [0;1] by the product of observations in each group]. The original paper by Mann and Whitney (1947) contains a proof for approximate normality and the U statistic is approximately Normal distributed even for small samples (~20). In that sense, the AUC could be assumed to be approximately Normal distributed.

To clarify some comments here, note that the test itself does not make any assumption on the distribution of the scores (usually the probability predictions from the model) (i.e. the test is non-parametric).

However, the exact variance of the AUC can only be calculated based on the distribution of the scores (Cortes & Mohri 2005), which typically do not follow a known distribution. There seem to be different suggestions on how to calculate a useful confidence interval, see Cortes & Mohri (2005) for a summary and their (long) formula in Corollary 1 ([4]):

$$ \sigma^2(AUC) = \frac{(m+n+1)(m+n)(m+n−1)T((m+n−2)Z_4−(2m−n+3k−10)Z_3)} {72m^2n^2} + \frac{(m+n+1)(m+n)T(m^2−nm+3km−5m+2k^2−nk+12−9k)Z_2}{48m^2n^2} − \frac{(m+n+1)^2(m−n)^4 Z_1^2} { 16m^2n^2} − \frac{(m+n+1)Q_1 Z_1}{72m^2n^2} + \frac{kQ_0}{144m^2n^2} $$ with: $$ Z_i = \frac{ \sum^{k−i}_{x=0} \left(\begin{smallmatrix} m+n+1−i \\ x \end{smallmatrix}\right) } { \sum^k_{x=0} \left(\begin{smallmatrix} m+n+1 \\ x \end{smallmatrix}\right) } $$

$$ T = 3((m − n)^2 + m + n) + 2 $$

$$ Q_0 = (m + n + 1)T k^2 + ((−3n^2 + 3mn + 3m + 1)T − 12(3mn + m + n) − 8)k + (−3m^2 +7m + 10n + 3nm + 10)T − 4(3mn + m + n + 1) $$

$$ Q_1 = T k^3 + 3(m − 1)T k^2 + ((−3n^2 + 3mn − 3m + 8)T − 6(6mn + m + n))k + (−3m^2 +7(m + n) + 3mn)T − 2(6mn + m + n) $$


Cortes, C., & Mohri, M. (2005). Confidence intervals for the area under the ROC curve. In Advances in neural information processing systems (pp. 305-312). https://cs.nyu.edu/~mohri/pub/area.pdf


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.