6
$\begingroup$

I found an interesting way to calculate a confidence interval for the Gini and respectively AUC coefficient for credit risk scoring.

Question: Can anyone explain me, why the sum

$$ AUC = \frac{1}{n \cdot m}\sum_{i=1}^n \sum_{j=1}^m S(X_i,y_j), \text{with} $$ $$ S(x_i,y_j) = \begin{cases} 1 & \text{if } x_i > y_j \\ 0.5 & \text{if } x_i = y_j \\ 0 & \text{if } x_i < y_j \end{cases} $$ has got the standard deviation $$ SE(AUC)= $$ $$\sqrt{\frac{AUC(1-AUC) + (m-1)\left(\frac{AUC}{2-AUC}- AUC^2\right) + (n-1)\left(\frac{2AUC^2}{1+AUC}-AUC^2\right)}{n \cdot m}} $$ and especially why the AUC is accepted as standard normal distributed?

In the book "Kreditrisikomessung" written by Henking, Bluhm and Fahrmeier (ISBN-10 3-540-32145-4 on page 223) is then a confidence interval given by

$$ AUC \pm z_{\alpha/2} SE(AUC) $$

$\endgroup$
  • $\begingroup$ This book amazon.com/Continuous-Chapman-Monographs-Statistics-Probability/… presents several alternative formulations of standard errors and CIs for ROC AUC, with references to the primary literature. It really is an indispensable resource if you are interested in these topics. $\endgroup$ – Sycorax Mar 8 '18 at 21:23
  • $\begingroup$ I can't access an English version of the reference you mention and my German is bad. Could you expand a bit on where the standard normal distribution is accepted? My understanding was that the SE calculation you show assumes negative exponential. The justification for this is a little wishy washy (it gave the easiest equations and was the most conservative) $\endgroup$ – ReneBt Mar 15 '18 at 12:36
  • $\begingroup$ Thank you very much ReneBt. Your answer is really helpful, but I'm still wondering about the assumption of normal distribution. Henking, Bluhm and Fahrmeir get their information from this document I think: citeseerx.ist.psu.edu/viewdoc/… $\endgroup$ – T. Beige Mar 19 '18 at 13:11
2
$\begingroup$

Assumptions addressed

The paper that proposed your formula (Hanely and MacNeil 1982) explicitly states that a key assumption is that the ratings are derived from a continuous scale that does not produce ‘ties’.

A typical explanation of "AUC is the probability that a sample randomly taken from the positive cases will rank higher than a randomly chosen negative case". If you have a tie then the positive case is not ranked higher and so S should = 0 to fit this explanation. If you include 0.5 for ties then the definition should be "AUC is the probability that a sample randomly taken from the positive cases will rank equal or higher than a randomly chosen negative case" This means that 1-AUC is the probability that a sample randomly taken from the positive cases will rank lower than a randomly chosen negative case (for 0.5 when tied) or equal or lower for 0 when tied case.

If you compare tie behaviour of S= 0 with S = 0.5 then AUC and 1-AUC will swap values everytime a tie occurs, so if there are a large proportion of ties the SE will diverge significantly between the two definitions. Since using S=0.5 when tied increases AUC it will therefore lead to a decrease in the calculated SE. I can see why Hanley and MacNeil ignored ties, maybe other readers will know of another source that explicitly details how ties impact on the SE calculation and can fill in that gap better.

Meaning of Elements in the equation

I'll include all elements for completeness, even the obvious ones.

AUC(1-AUC) is self explanatory, the AUC times its inverse. It is at a maximum when AUC = 0.5, i.e. it is a squared value and becomes smaller as AUC deviates from 0.5. Since AUC = 0.5 is equivalent to random chance then you would expect there to be large uncertainty at this value and for it to diminish as the value increases.

M is the number of positive cases

Hanley and MacNeil had compared gaussian, gamma and negative exponential distribution assumptions and chose the latter as it was the most conservative of that set and also provided the easiest terms to use in the equation. This assumption is the basis of using AUC/(2-AUC) and 2AUC^2 / (1+AUC) in the formula.

AUC/(2-AUC) is the probability of ranking two randomly chosen positive sample higher than a negative one based on an underlying assumption of a negative exponential distribution

Thus $$ \frac{AUC}{2-AUC}- AUC^2 $$ is the difference between the probability of two positive samples being ranked higher and the square of the probability of one sample being ranked higher than a random negative one.

N is the number of negative cases

2AUC^2 / (1+AUC) is the probability of ranking one randomly chosen positive samples higher than two randomly chosen negative ones based on an underlying assumption of negative exponential distribution

Thus $$ \frac{2AUC^2}{1+AUC}-AUC^2\ $$ is the difference between the probability of one positive samples being ranked higher than two negative ones and the square of the probability of one positive sample being ranked higher than one random negative sample.

Basically the equation is taking second order effects into account as well as first order effects (the SE for a Bernoulli only uses first order effects)

The SE equation you quote assumes negative exponential rather than standard normal distribution. I’ll add a comment to the question to get more details on what you mean and what sources. Without understanding the background to this I can’t really address it.

$\endgroup$
  • $\begingroup$ My interpretation of the question is more along the lines of asking for a derivation of SE(AUC), i.e. can you show the derivation of the general formula using this and this. And then Hanley & McNeil simplify this by assuming exponential distributions for X and Y. Also, can you show why AUC/(2-AUC) = the probability of ranking two randomly chosen positive sample higher than a negative one assuming a exponential distribution for X and Y? (Same for the other expression). $\endgroup$ – user5594 Mar 15 '18 at 13:38
  • $\begingroup$ I'll try and put something more rigorous together. In saying that, I'm not a big fan of AUC although I do reluctantly use it as that is what many scientists are comfortable with. It is a severely limited statistic that I only see as useful for coarse grain filtering of competing models in the model building phase. It completely ignores the subtleties of how a test interacts with the needs in the application space, so I've not been inclined to put too much weight on it so I've always accepted Hanley and MacNeil without too much probing. Ultimately AUC is sidelined when it really matters. $\endgroup$ – ReneBt Mar 16 '18 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.