Skip to main content
2 of 4
added 44 characters in body
Richard Hardy
  • 69.5k
  • 13
  • 126
  • 278

Question 1

Given a threshold* $t$, model 1 has lower estimated expected loss than model 2 if the corresponding ROC point of model 1 dominates** the ROC point of model 2. Here is why.

Let the confusion matrix corresponding to a particular threshold $t$ be $$ \text{Conf}_t=\begin{pmatrix} j_t & k_t\\ l_t & m_t \end{pmatrix} $$ with predicted classes in rows (row 1 ~ class 0, row 2 ~ class 1) and actual classes in columns (column 1 ~ class 0, column 2 ~ class 1). Concretely, $$ \text{Conf}_t=\begin{pmatrix} \#\{{\hat Y=0|Y=0\}}_t & \#\{{\hat Y=0|Y=1\}}_t \\ \#\{{\hat Y=1|Y=0\}}_t & \#\{{\hat Y=1|Y=1\}}_t \end{pmatrix} $$ with $\#$ counting the number of elements that satisfy the condition. We will later add a subscript 1 for model 1 and 2 for model 2.

For any given sample, the number of actual zeros $j_t+l_t$ and the number of actual unities $k_t+m_t$ are fixed at $r$ and $s$, respectively: \begin{aligned} j_t+l_t &= r \quad \text{and} \\ k_t+m_t &= s. \end{aligned} We will make use of the latter equality in a subsequent step. Let us also define the sample size $$ n:=j_t+k_t+l_t+m_t. $$

The estimated expected loss of a model is \begin{aligned} \hat{\mathbb{E}}(L) &= \frac{1}{n}\big[ak_{t}+bl_{t}\big] \\ &= \frac{1}{n}\big[a(s-m_{t})+bl_{t}\big]. \end{aligned} Explicitly, the estimated expected losses of models 1 and 2 are \begin{aligned} \hat{\mathbb{E}}(L_1) &= \frac{1}{n}\big[a(s-m_{1t})+bl_{1t}\big] \quad \text{and} \\ \hat{\mathbb{E}}(L_2) &= \frac{1}{n}\big[a(s-m_{2t})+bl_{2t}\big]. \end{aligned}

The ROC points (specific to the threshold $t$) of models 1 and 2 have coordinates $(l_{1t},m_{1t})$ and $(l_{2t},m_{2t})$, respectively. If the former point dominates the latter point, we have $l_{1t}\leq l_{2t}$ and $m_{1t}\geq m_{2t}$ and at least one of the two inequalities is strict.

What does this imply regarding $\hat{\mathbb{E}}(L_1)$ vs. $\hat{\mathbb{E}}(L_2)$? Since $a,b>0$ and $s-m_{1t},s-m_{2t}\geq0$, then looking at the formulas above we immediately see that $\hat{\mathbb{E}}(L_1)<\hat{\mathbb{E}}(L_2)$. Thus if the ROC point of model 1 dominates the ROC point of model 2, model 1 has lower estimated expected loss than model 2.

*The relevant threshold would be the optimal one. **{ Is above and to the left } OR { is above and not to the right } OR { is to the left and not below }.

Question 2: still open.

Richard Hardy
  • 69.5k
  • 13
  • 126
  • 278