The area-under-the-receiver-operating-characteristic-curve (AUROC) can be derived from the Bayes Minimum Risk. The derivation requires the assumption that the exact costs are unknown but follow a distribution that depends on the score distribution (Hand, 2009). The connection between costs and the model scores for the AUC is interesting, since the evaluation criterion should not depend on characteristics of the particular model. I have been unable to follow the actual derivation of the result stated briefly in Hand & Till (2001), Hand (2009) and Krzanowski & Hand (2009).

Define model scores $s$, an optimal threshold given costs $T$ and the probability density function $f_c(t) = p(t|c)$ and cumulative distribution function $F_c(t) = p(x<t|c)$, where we have two classes $c \in \{0;1\}$. Given a particular assumption on the weight, from the Bayes minimum risk we get this expression of the expected minimum cost (Hand 2009, p.111):

$$ \int_{-\infty}^{\infty} {p(c=0)p(c=1)\{f_1(T)(1−F_0(T))+f_0(T)F_1(T)}\}dT, $$ where $T$ is the risk optimal threshold.

I don't understand the step from that equation to the next:

$$ p(c=0)p(c=1) \{ \int_{-\infty}^{\infty}\int_{T}^{\infty} f_1(T)f_0(s)dsdT+ \int_{-\infty}^{\infty}\int_{-\infty}^{s} f_1(T)f_0(s)dTds \}. $$

I do understand that

  • we pull out $p(c=0)p(c=1)$, which are constant
  • the cdf $F_0(T) = \int_{-\infty}^{T} f_0(s)ds$, so $1-F_0(T) = \int_{T}^{\infty} f_0(s)ds$, because the pdf integrates to 1, so the left side seems clear.

On the right hand, I see that the order of integrations is switched ($dTds$) and that the conditionals on $p(T|\cdot)$ and $p(s|\cdot)$ change. How does this work?

My best guess starts with

$$ \int_{-\infty}^{\infty} f_0(T)F_1(T) = \int_{-\infty}^{\infty} p(T|c=0) \int_{-\infty}^{T} p(s|c=1) = \int_{-\infty}^{\infty} \int_{-\infty}^{T} \frac{p(T,c=0)}{p(c=0)} \frac{p(s,c=1)}{p(c=1)} $$

but then I am stuck.

This question is related to this question on the probabilistic interpretation of the AUC, but is concerned with the derivation of the AUC from the Bayes Minimum Risk.


Hand, D. J., & Till, R. J. (2001). A simple generalisation of the area under the ROC curve for multiple class classification problems. Machine learning, 45(2), 171-186.
Hand, D. J. (2009). Measuring classifier performance: a coherent alternative to the area under the ROC curve. Machine learning, 77(1), 103-123.
Krzanowski, W. J., & Hand, D. J. (2009). ROC curves for continuous data. Chapman and Hall/CRC.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.