Explaining a step in deriving the cost ratio in ROC curve as a function of AUC

Question

On this paper by David Hand the misclassification costs are $c_0$ and $c_1$ with the ratio $c$ defined as $c=c_0/(c_0+c_1).$ The threshold considered optimal is

$$T(c)=\underset{t}{\text{arg min}}\left\{ c\pi_0(1-F_0(t)) + (1-c) \pi_1 F_1(t) \right\}$$

with $t$ being any threshold in the scores, and $1=\pi_0+\pi_1$ the fractions of diseased and healthy individuals in the population. $F_0$ is the distribution of diseased, and $F_1$ the distribution of healthy individuals. Cases are considered class $0.$

Assuming a one-to-one relationship of $c$ with $T$ and differentiating, he gets the following result (equation (5)):

$$c= \color{red}{\Pr(1\vert T)} = \pi_1 f_1(T) / \left\{ \pi_0f_0(T) + \pi_1f_1(T)\right\}$$

where $\Pr(1\vert T)$ is the conditional probability of belonging to class $1,$ given the score $T.$

I follow everything in here, except for the part in red. It may be a very basic misunderstanding but shouldn't the conditional probability be calculated from a ratio of cdf's, as opposed to pdf's?

I presume the answer is in considering the scores $s(x)$ in the logistic regression as a monotonically increasing transformation of $\Pr(1 \vert x).$

---

Here is an unsuccessful attempt at hacking this:

install.packages('pROC')
install.packages('ROCR')
install.packages('Epi')
library(pROC)
library(ROCR)
library(Epi)

set.seed(561)

cost0 = 1   # Cost of mis-classifying a normal as having cancer in million $
cost1 = 10   # Cost of mis-classifying a cancer patient as normal (death?)

b = cost0 + cost1
c = cost0/(b)

n = 7000    # Total cases
pi0 =.8     # Percentage of normal
pi1 =.2     # Percentage of disease

# Actual values of the test for normals and disease (D higher test values)
testA_Normals = rnorm(n*pi0, mean=3, sd=1)
testA_Sick = rnorm(n*pi1, 6, 1)

# Determining a threshold based on cost 
# arg t min {Loss = cost0 * (1 - pnorm(t,3,1)) * pi0 + 
#            cost1 * pnorm(t,6,1) * pi1}

t = seq(0,10,0.0001)
loss <- cost0 * (1 - pnorm(t,3,1)) * pi0 + cost1 * pnorm(t,6,1) * pi1
Threshold = data.frame(t,loss)[which(loss==min(loss)),]$t

hist(testA_Normals,border=F, xlim=c(0,10))
hist(testA_Sick,col=2,border=F, add=T)

abline(v=Threshold)

Comparing the 3 equalities in the equation:

c
pi1 * dnorm(Threshold,6,1) / (pi0 * dnorm(Threshold,3,1) + pi1 * dnorm(Threshold,6,1))
#P(1|T) = P(T|1) * P(1) / [P(T|0) * P(0) + P(T|1) * P(1)]
(pnorm(Threshold,6,1,F)*pi1)/(pnorm(Threshold,6,1,F)*pi1+
                            pnorm(Threshold,3,1,F)*pi0)

0.0909090909090909
0.0909165896894187
0.6749524!!!!!(***)

$(***) \text{Edit}:$ After getting help from a very reliable source (unnamed lest I misquote) the thinking behind $\Pr(1\vert T)$ is not $\Pr(1\vert s \geq T),$ which would be what my code would suggest, but rather $\Pr(1\vert s=T),$ or $$\Tiny\lim_{d \to 0}\frac{d \pi_1 f_1(T)}{ d \pi_1 f_1(T) + d \pi_0 f_0(T) }= \lim_{d \to 0} \Pr\left(1\Big\vert -\frac d 2 +T +\frac d 2\right). $$

Answer 1

Given decision rule

When Hypothesis $\mathsf H_0$ is true (an event that occurs with probability $\pi_0$), the decision variable $X$ exceeds the threshold $t$ with probability $(1-F_0(t))$ (and so a false alarm occurs) and the cost incurred is $c_0$.

When Hypothesis $\mathsf H_1$ is true (an event that occurs with probability $\pi_1$), the decision variable $X$ is smaller than the threshold $t$ with probability $F_1(t)$ (and so a missed detection occurs) and the cost incurred is $c_1$.

Thus, the average cost or expected cost of each decision is \begin{align} \text{average cost} &= c_0\pi_0(1-F_0(t)) + c_1\pi_1F_1(t)\\\ &= (c_0 + c_1)\left[\frac{c_0}{c_0 + c_1}\pi_0(1-F_0(t)) + \frac{c_1}{c_0 + c_1}\pi_1F_1(t)\right]\\ &= (c_0 + c_1)\big[c\pi_0(1-F_0(t)) + (1-c)\pi_1F_1(t)\big]. \end{align} The value of $t$ that minimizes the average cost is thus $$T = \underset{t}{\arg \min}\big[c\pi_0(1-F_0(t)) + (1-c)\pi_1F_1(t)\big],\tag{1}$$ and the minimum average cost that this decision rule can achieve is $$\text{minimum average cost}=(c_0 + c_1)\big[c\pi_0(1-F_0(T)) + (1-c)\pi_1F_1(T)\big]. \tag{2}$$

Note, however, that this minimality of the average cost is only among all decision rules of the form

If $X > t$, the decision is that $\mathsf H_1$ occurred.
If $X \leq t$, the decision is that $\mathsf H_0$ occurred.

Other decision rules may well achieve smaller average costs than $(2)$, and we discuss these below.

---

Optimal minimum-average-cost decision rule

The optimal minimum-expected-cost decision rule is the one that compares the likelihood ratio $\displaystyle\Lambda(X) = \frac{f_1(X)}{f_0(X)}$ to the threshold $\displaystyle\frac{c_0\pi_0}{c_1\pi_1}$ and decides that $\mathsf H_0$ or $\mathsf H_1$ occurred according as $\Lambda(X)$ is less than or equal to the threshold or is larger than the threshold. Thus, the real line can be partitioned into sets $\Gamma_0$ and $\Gamma_1$ defined as \begin{align} \Gamma_0 &= \big\{X \in \Gamma_0 \implies \textit{decision }\text{is that } \mathsf H_0~\text{occurred}\big\}\\ &= \left\{x\in \mathbb R\colon \Lambda(x) \leq \frac{c_0\pi_0}{c_1\pi_1}\right\}\\ \Gamma_1 &= \big\{X \in \Gamma_1 \implies \textit{decision }\text{is that } \mathsf H_1~\text{occurred}\big\}\\ &= \left\{x\in \mathbb R\colon \Lambda(x) > \frac{c_0\pi_0}{c_1\pi_1}\right\} \end{align} where $\Gamma_0$ and $\Gamma_1$ are not necessarily the sets $\left\{x \leq T\right\}$ and $\left\{x > T\right\}$ discussed previously. The optimal minimum-average-cost decision has a cost of $$\text{minimum average cost}=(c_0 + c_1)\big[c\pi_0\Pr\{X \in \Gamma_1\mid \mathsf H_0\} + (1-c)\pi_1\Pr\{X \in \Gamma_0\mid \mathsf H_1\}\big]. \tag{3}$$

If the likelihood ratio is a monotone increasing function of its argument,

then $\Gamma_0$ and $\Gamma_1$ are found to be of the form $\left\{x \leq T^*\right\}$ and $\left\{x > T^*\right\}$ and $(3)$ simplifies to \begin{align} \text{minimum average cost}&=(c_0 + c_1)\big[c\pi_0\Pr\{X > T^*\mid \mathsf H_0\} + (1-c)\pi_1\Pr\{X \leq T^*\mid \mathsf H_1\}\big]\\ &= (c_0 + c_1)\big[c\pi_0(1-F_0(T^*)) + (1-c)\pi_1F_1(T^*)\big]. \tag{4} \end{align} A little thought shows that $T^*$ necessarily must the same as $T$ in $(1)$. But there is more information to be obtained from $(4)$ because now we have a different description of the value of $T^*$.

$T^*$ is the number such that $\Lambda(T^*)$ equals $\displaystyle\frac{c_0\pi_0}{c_1\pi_1}$.

From $\displaystyle\Lambda(T^*) = \frac{f_1(T^*)}{f_0(T^*)} = \frac{c_0\pi_0}{c_1\pi_1}$, we get (with some straightforward algebra and the claim that $T^*$ equals $T$) that $$c =\frac{c_0}{c_0+c_1} = \frac{\pi_1f_1(T^*)}{\pi_0f_0(T^*)+\pi_1f_1(T^*)} = \frac{\pi_1f_1(T)}{\pi_0f_0(T)+\pi_1f_1(T)}$$ whose derivation is what puzzled the OP.

Finally, let's turn to the claim that $c$ also equals $\Pr(1\mid T)$. Let $Y$ be a Bernoulli random variable such that $Y=1$ whenever $\mathsf H_1$ occurs while $Y=0$ when $\mathsf H_0$ occurs. Thus we have that for $i=0,1$, $f_{X\mid Y=i}(x) := f_i(x)$. Now, $X$ and $Y$ can't enjoy a joint density function because $Y$ is not a continuous random variable, and if we want to visualize the $x$-$y$ plane, then we have two (weighted) line densities $\pi_0f_0(x)$ and $\pi_1f_1(x)$ along the lines $y=0$ and $y=1$ in the $x$-$y$ plane. What is the unconditional density of $X$? Well, at $X=x$, the unconditional density of $X$ has value $$f_X(x) = \pi_0f_0(x)+\pi_1f_1(x).\tag{5}$$ Turning matters around, what is the distribution of the Bernoulli random variable $Y$ conditioned on $X=x$? Well, when $X=x$, $Y$ takes on values $0$ and $1$ with respective probabilities \begin{align}\Pr(Y=0\mid X=x) &= \frac{\pi_0f_0(x)}{\pi_0f_0(x)+\pi_1f_1(x)}\tag{6}\\ \Pr(Y=1\mid X=x) &= \frac{\pi_1f_1(x)}{\pi_0f_0(x)+\pi_1f_1(x)}\tag{7} \end{align} which shows that $c$ equals $\Pr(Y=1\mid X=T)$ which the paper that the OP is reading writes as $\Pr(1|T)$. That's machine learning lingo for you.... But are $(6)$ and $(7)$ plausible values for the conditional pdf of $Y$? Well, for $i=0,1$, we can find the unconditional probability that $Y=i$ by multiplying the conditional probability $\Pr(Y=i\mid X=x)$ by the pdf of $X$ and integrating which gives us \begin{align} \Pr(Y=i) &= \int_{-\infty}^\infty \Pr(Y=i\mid X=x)\cdot f_X(x) \,\mathrm dx\\ &= \int_{-\infty}^\infty \left.\left.\frac{\pi_if_i(x)}{\pi_0f_0(x)+\pi_1f_1(x)} \cdot \right(\pi_0f_0(x)+\pi_1f_1(x)\right) \,\mathrm dx\\ &= \int_{-\infty}^\infty \pi_if_i(x) \,\mathrm dx\\ &= \pi_i \end{align} which I hope adds a touch of artistic verisimilitude to an otherwise bald and unconvincing narrative.