9
$\begingroup$

I am following this paper, from page 47 on.

I have some model from which I can construct ROC and calculate its $AUC$. 95% confidence interval will be $[AUC - x, AUC + x]$. How do I interpret it? I assume that if lower bound of interval is higher than 0.5 then I can conclude that my model is better than random one. What confuses me is that $AUC$ is in the middle of interval so it will always be inside CI. I think that maybe if my model was applied to some different observation, I would be 95% sure that its $AUC$ fit into CI. Am I right?

Thanks.

$\endgroup$

2 Answers 2

13
$\begingroup$

A confidence interval is an interval-estimate for some true value of a parameter. Let us (as an example) start with e.g. a confidence interval for the mean of a normal distribution and then move on to ROC and AUC so that one sees the analogy.

Assume that you have a random normal variable $X \sim N(\mu;\sigma)$. Where $\mu$ is the unknown population mean and, to keep it simple, let us assume that $\sigma$ is known.

We now draw a sample of size $n$ from the distribution of X, i.e. we get a sample $x_1, x_2, \dots x_n$. The goal is to have an idea about the unknown $\mu$ using the sample drawn. It is well known that the arithemetic average $\bar{x}=\frac{1}{n}\sum_i x_i$ is an unbiased (point) estimator for (the unknown) $\mu$ and that $[\bar{x}-1.96\frac{\sigma}{\sqrt{n}};\bar{x}+1.96\frac{\sigma}{\sqrt{n}}]$ is a $95\%$ confidence interval for (the unknown) $\mu$.

If we draw another sample $y_1, \dots , y_n$ from the distribtion of $X$ then, in the same way we will find another confidence interval for the (unknown) $\mu$ as $[\bar{y}-1.96\frac{\sigma}{\sqrt{n}};\bar{y}+1.96\frac{\sigma}{\sqrt{n}}]$.

So each time we draw a sample of size $n$ from the distribution of $X$, we find a confidence interval for the (unknown) $\mu$ and all these intervals will be different. The fact that it is a $95\%$ confidence interval means that, if we draw an 'infinite' number of samples of size $n$ from the distribution of $X$, and for each of these samples we compute the $95\%$ confidence interval, then $95\%$ of all these intervals (one interval for each sample) will contain the unknown $\mu$. (so sometimes , namely $5\%$ of the intervals, such an interval will not contain the unknown $\mu$, so sometimes you have bad luck.)

The same holds for the AUC, when you compute the AUC, you compute it from a sample, in other words what you compute is an estimate for the true unknown AUC. Similarly you can, for the sample that you have, compute a confidence interval for the true but unknown AUC. If you were able to draw an infinite number of samples, and for each sample obtained compute the confidence interval for the true AUC, then $95\%$ of these computed intervals would contain the true but unknown AUC.

Note that the interval is random, because it is computed from a random sample. The true AUC is not random, it is some unknown property of your population.

Unfortunately you can not draw an infinite number of samples, most of the time you have only one sample, so you will have to do it with one interval, but you are rather confident ($95\%$ of the so computed intervals will contain the true unknown AUC) that this interval will contain the true AUC. And yes, if the lower border of the interval is higer than 0.5 then you can be rather confident that your model is not the random model, but, as above, you may also have had bad luck with the sample.

$\endgroup$
6
  • 1
    $\begingroup$ Great answer, thanks a lot! I thought about my computed AUC as a true AUC rather than AUC of one sample. $\endgroup$
    – sruzic
    Commented Aug 14, 2015 at 21:25
  • 1
    $\begingroup$ @sruzic: don't thank me, just vote for the answer if you like it $\endgroup$
    – user83346
    Commented Aug 14, 2015 at 21:37
  • 2
    $\begingroup$ I would but I do not have 15 rep yet .. :/ $\endgroup$
    – sruzic
    Commented Aug 14, 2015 at 21:38
  • 1
    $\begingroup$ @sruzic: no problem, I am glad that I made it a bit clearer :-) $\endgroup$
    – user83346
    Commented Aug 14, 2015 at 21:41
  • 3
    $\begingroup$ @f coppens: Here is your upvote ;) $\endgroup$
    – sruzic
    Commented Aug 22, 2015 at 9:20
2
$\begingroup$

Probably the best interpretation would be in terms of the so-called $c$ statistic, which turns out to equal the area under the ROC curve. That is, if you are trying to predict some response $Y$ (which is often binary) using a score $X$, then the $c$ statistic is defined as $P(X^\prime > X \mid Y^\prime > Y)$, where $X^\prime$ and $Y^\prime$ are independent copies of $X$ and $Y$.

You would then be $95\%$ confident that the "true" value of this conditional probability lies within the specified interval. This would allow you to somewhat more formally reject the claim that your model is no better than random if the lower bound is above $1/2$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.