# Explaining a step in deriving the cost ratio in ROC curve as a function of AUC

On this paper by David Hand the misclassification costs are $$c_0$$ and $$c_1$$ with the ratio $$c$$ defined as $$c=c_0/(c_0+c_1).$$ The threshold considered optimal is

$$T(c)=\underset{t}{\text{arg min}}\left\{ c\pi_0(1-F_0(t)) + (1-c) \pi_1 F_1(t) \right\}$$

with $$t$$ being any threshold in the scores, and $$1=\pi_0+\pi_1$$ the fractions of diseased and healthy individuals in the population. $$F_0$$ is the distribution of diseased, and $$F_1$$ the distribution of healthy individuals. Cases are considered class $$0.$$

Assuming a one-to-one relationship of $$c$$ with $$T$$ and differentiating, he gets the following result (equation (5)):

$$c= \color{red}{\Pr(1\vert T)} = \pi_1 f_1(T) / \left\{ \pi_0f_0(T) + \pi_1f_1(T)\right\}$$

where $$\Pr(1\vert T)$$ is the conditional probability of belonging to class $$1,$$ given the score $$T.$$

I follow everything in here, except for the part in red. It may be a very basic misunderstanding but shouldn't the conditional probability be calculated from a ratio of cdf's, as opposed to pdf's?

I presume the answer is in considering the scores $$s(x)$$ in the logistic regression as a monotonically increasing transformation of $$\Pr(1 \vert x).$$

Here is an unsuccessful attempt at hacking this:

install.packages('pROC')
install.packages('ROCR')
install.packages('Epi')
library(pROC)
library(ROCR)
library(Epi)

set.seed(561)

cost0 = 1   # Cost of mis-classifying a normal as having cancer in million $cost1 = 10 # Cost of mis-classifying a cancer patient as normal (death?) b = cost0 + cost1 c = cost0/(b) n = 7000 # Total cases pi0 =.8 # Percentage of normal pi1 =.2 # Percentage of disease # Actual values of the test for normals and disease (D higher test values) testA_Normals = rnorm(n*pi0, mean=3, sd=1) testA_Sick = rnorm(n*pi1, 6, 1) # Determining a threshold based on cost # arg t min {Loss = cost0 * (1 - pnorm(t,3,1)) * pi0 + # cost1 * pnorm(t,6,1) * pi1} t = seq(0,10,0.0001) loss <- cost0 * (1 - pnorm(t,3,1)) * pi0 + cost1 * pnorm(t,6,1) * pi1 Threshold = data.frame(t,loss)[which(loss==min(loss)),]$t

hist(testA_Normals,border=F, xlim=c(0,10))

abline(v=Threshold)


Comparing the 3 equalities in the equation:

c
pi1 * dnorm(Threshold,6,1) / (pi0 * dnorm(Threshold,3,1) + pi1 * dnorm(Threshold,6,1))
#P(1|T) = P(T|1) * P(1) / [P(T|0) * P(0) + P(T|1) * P(1)]
(pnorm(Threshold,6,1,F)*pi1)/(pnorm(Threshold,6,1,F)*pi1+
pnorm(Threshold,3,1,F)*pi0)

0.0909090909090909
0.0909165896894187
0.6749524!!!!!(***)


$$(***) \text{Edit}:$$ After getting help from a very reliable source (unnamed lest I misquote) the thinking behind $$\Pr(1\vert T)$$ is not $$\Pr(1\vert s \geq T),$$ which would be what my code would suggest, but rather $$\Pr(1\vert s=T),$$ or $$\Tiny\lim_{d \to 0}\frac{d \pi_1 f_1(T)}{ d \pi_1 f_1(T) + d \pi_0 f_0(T) }= \lim_{d \to 0} \Pr\left(1\Big\vert -\frac d 2 +T +\frac d 2\right).$$

Given decision rule

When Hypothesis $$\mathsf H_0$$ is true (an event that occurs with probability $$\pi_0$$), the decision variable $$X$$ exceeds the threshold $$t$$ with probability $$(1-F_0(t))$$ (and so a false alarm occurs) and the cost incurred is $$c_0$$.

When Hypothesis $$\mathsf H_1$$ is true (an event that occurs with probability $$\pi_1$$), the decision variable $$X$$ is smaller than the threshold $$t$$ with probability $$F_1(t)$$ (and so a missed detection occurs) and the cost incurred is $$c_1$$.

Thus, the average cost or expected cost of each decision is \begin{align} \text{average cost} &= c_0\pi_0(1-F_0(t)) + c_1\pi_1F_1(t)\\\ &= (c_0 + c_1)\left[\frac{c_0}{c_0 + c_1}\pi_0(1-F_0(t)) + \frac{c_1}{c_0 + c_1}\pi_1F_1(t)\right]\\ &= (c_0 + c_1)\big[c\pi_0(1-F_0(t)) + (1-c)\pi_1F_1(t)\big]. \end{align} The value of $$t$$ that minimizes the average cost is thus $$T = \underset{t}{\arg \min}\big[c\pi_0(1-F_0(t)) + (1-c)\pi_1F_1(t)\big],\tag{1}$$ and the minimum average cost that this decision rule can achieve is $$\text{minimum average cost}=(c_0 + c_1)\big[c\pi_0(1-F_0(T)) + (1-c)\pi_1F_1(T)\big]. \tag{2}$$

Note, however, that this minimality of the average cost is only among all decision rules of the form

If $$X > t$$, the decision is that $$\mathsf H_1$$ occurred.
If $$X \leq t$$, the decision is that $$\mathsf H_0$$ occurred.

Other decision rules may well achieve smaller average costs than $$(2)$$, and we discuss these below.

Optimal minimum-average-cost decision rule

The optimal minimum-expected-cost decision rule is the one that compares the likelihood ratio $$\displaystyle\Lambda(X) = \frac{f_1(X)}{f_0(X)}$$ to the threshold $$\displaystyle\frac{c_0\pi_0}{c_1\pi_1}$$ and decides that $$\mathsf H_0$$ or $$\mathsf H_1$$ occurred according as $$\Lambda(X)$$ is less than or equal to the threshold or is larger than the threshold. Thus, the real line can be partitioned into sets $$\Gamma_0$$ and $$\Gamma_1$$ defined as \begin{align} \Gamma_0 &= \big\{X \in \Gamma_0 \implies \textit{decision }\text{is that } \mathsf H_0~\text{occurred}\big\}\\ &= \left\{x\in \mathbb R\colon \Lambda(x) \leq \frac{c_0\pi_0}{c_1\pi_1}\right\}\\ \Gamma_1 &= \big\{X \in \Gamma_1 \implies \textit{decision }\text{is that } \mathsf H_1~\text{occurred}\big\}\\ &= \left\{x\in \mathbb R\colon \Lambda(x) > \frac{c_0\pi_0}{c_1\pi_1}\right\} \end{align} where $$\Gamma_0$$ and $$\Gamma_1$$ are not necessarily the sets $$\left\{x \leq T\right\}$$ and $$\left\{x > T\right\}$$ discussed previously. The optimal minimum-average-cost decision has a cost of $$\text{minimum average cost}=(c_0 + c_1)\big[c\pi_0\Pr\{X \in \Gamma_1\mid \mathsf H_0\} + (1-c)\pi_1\Pr\{X \in \Gamma_0\mid \mathsf H_1\}\big]. \tag{3}$$

If the likelihood ratio is a monotone increasing function of its argument,

then $$\Gamma_0$$ and $$\Gamma_1$$ are found to be of the form $$\left\{x \leq T^*\right\}$$ and $$\left\{x > T^*\right\}$$ and $$(3)$$ simplifies to \begin{align} \text{minimum average cost}&=(c_0 + c_1)\big[c\pi_0\Pr\{X > T^*\mid \mathsf H_0\} + (1-c)\pi_1\Pr\{X \leq T^*\mid \mathsf H_1\}\big]\\ &= (c_0 + c_1)\big[c\pi_0(1-F_0(T^*)) + (1-c)\pi_1F_1(T^*)\big]. \tag{4} \end{align} A little thought shows that $$T^*$$ necessarily must the same as $$T$$ in $$(1)$$. But there is more information to be obtained from $$(4)$$ because now we have a different description of the value of $$T^*$$.

$$T^*$$ is the number such that $$\Lambda(T^*)$$ equals $$\displaystyle\frac{c_0\pi_0}{c_1\pi_1}$$.

From $$\displaystyle\Lambda(T^*) = \frac{f_1(T^*)}{f_0(T^*)} = \frac{c_0\pi_0}{c_1\pi_1}$$, we get (with some straightforward algebra and the claim that $$T^*$$ equals $$T$$) that $$c =\frac{c_0}{c_0+c_1} = \frac{\pi_1f_1(T^*)}{\pi_0f_0(T^*)+\pi_1f_1(T^*)} = \frac{\pi_1f_1(T)}{\pi_0f_0(T)+\pi_1f_1(T)}$$ whose derivation is what puzzled the OP.

Finally, let's turn to the claim that $$c$$ also equals $$\Pr(1\mid T)$$. Let $$Y$$ be a Bernoulli random variable such that $$Y=1$$ whenever $$\mathsf H_1$$ occurs while $$Y=0$$ when $$\mathsf H_0$$ occurs. Thus we have that for $$i=0,1$$, $$f_{X\mid Y=i}(x) := f_i(x)$$. Now, $$X$$ and $$Y$$ can't enjoy a joint density function because $$Y$$ is not a continuous random variable, and if we want to visualize the $$x$$-$$y$$ plane, then we have two (weighted) line densities $$\pi_0f_0(x)$$ and $$\pi_1f_1(x)$$ along the lines $$y=0$$ and $$y=1$$ in the $$x$$-$$y$$ plane. What is the unconditional density of $$X$$? Well, at $$X=x$$, the unconditional density of $$X$$ has value $$f_X(x) = \pi_0f_0(x)+\pi_1f_1(x).\tag{5}$$ Turning matters around, what is the distribution of the Bernoulli random variable $$Y$$ conditioned on $$X=x$$? Well, when $$X=x$$, $$Y$$ takes on values $$0$$ and $$1$$ with respective probabilities \begin{align}\Pr(Y=0\mid X=x) &= \frac{\pi_0f_0(x)}{\pi_0f_0(x)+\pi_1f_1(x)}\tag{6}\\ \Pr(Y=1\mid X=x) &= \frac{\pi_1f_1(x)}{\pi_0f_0(x)+\pi_1f_1(x)}\tag{7} \end{align} which shows that $$c$$ equals $$\Pr(Y=1\mid X=T)$$ which the paper that the OP is reading writes as $$\Pr(1|T)$$. That's machine learning lingo for you.... But are $$(6)$$ and $$(7)$$ plausible values for the conditional pdf of $$Y$$? Well, for $$i=0,1$$, we can find the unconditional probability that $$Y=i$$ by multiplying the conditional probability $$\Pr(Y=i\mid X=x)$$ by the pdf of $$X$$ and integrating which gives us \begin{align} \Pr(Y=i) &= \int_{-\infty}^\infty \Pr(Y=i\mid X=x)\cdot f_X(x) \,\mathrm dx\\ &= \int_{-\infty}^\infty \left.\left.\frac{\pi_if_i(x)}{\pi_0f_0(x)+\pi_1f_1(x)} \cdot \right(\pi_0f_0(x)+\pi_1f_1(x)\right) \,\mathrm dx\\ &= \int_{-\infty}^\infty \pi_if_i(x) \,\mathrm dx\\ &= \pi_i \end{align} which I hope adds a touch of artistic verisimilitude to an otherwise bald and unconvincing narrative.

• This is a fantastic answer with magisterial expositional skills. Thank you! I am a bit concerned, though, that the part that puzzled me is not really the last equation in your post, but the fact that the last equation is equated, in the article quoted, to $\Pr(1\vert T)$ - the conditional probability. Yet, the numerator and denominator do not contain cdf's (as I would expect for probabilities to be calculated) - instead, there are pdf's (densities). In this regard, although I can reproduce the last equation on your post in my simulation, I can't equate it to $c=\Pr(1\vert T)$ in R. Commented Dec 22, 2020 at 23:13
• For context, I am ultimately aiming to understand the discrete implicit cost distribution that seems to vary from classifier to classifier in David Hand's article, and which is object of my related question. This is the plotROC(x, which=3) in the R package hmeasure. I figured understanding this step would make the rest easier, but maybe it is not needed... Commented Dec 22, 2020 at 23:31
• @AntoniParellada See revised answer. Commented Dec 22, 2020 at 23:56