Return to Answer

Question 1

Given a threshold* $t$, model 1 has lower estimated expected loss than model 2 if the corresponding ROC point of model 1 dominates** the ROC point of model 2. Here is why.

Let the confusion matrix corresponding to a particular threshold $t$ be $$ \text{Conf}_t=\begin{pmatrix} j_t & k_t\\ l_t & m_t \end{pmatrix} $$ with predicted classes in rows (row 1 ~ class 0, row 2 ~ class 1) and actual classes in columns (column 1 ~ class 0, column 2 ~ class 1). Concretely, $$ \text{Conf}_t=\begin{pmatrix} \#\{{\hat Y=0 \cap Y=0\}}_t & \#\{{\hat Y=0\cap Y=1\}}_t \\ \#\{{\hat Y=1\cap Y=0\}}_t & \#\{{\hat Y=1\cap Y=1\}}_t \end{pmatrix} $$ with $\#$ counting the number of elements that satisfy the condition. We will later add a subscript 1 for model 1 and 2 for model 2.

For any given sample, the number of actual zeros $j_t+l_t$ and the number of actual unities $k_t+m_t$ are fixed at $r$ and $s$, respectively: \begin{aligned} j_t+l_t &= r \quad \text{and} \\ k_t+m_t &= s. \end{aligned} We will make use of the latter equality in a subsequent step. Let us also define the sample size $$ n:=j_t+k_t+l_t+m_t. $$

The estimated expected loss of a model is \begin{aligned} \hat{\mathbb{E}}(L) &= \frac{1}{n}\big[ak_{t}+bl_{t}\big] \\ &= \frac{1}{n}\big[a(s-m_{t})+bl_{t}\big]. \end{aligned} Explicitly, the estimated expected losses of models 1 and 2 are \begin{aligned} \hat{\mathbb{E}}(L_1) &= \frac{1}{n}\big[a(s-m_{1t})+bl_{1t}\big] \quad \text{and} \\ \hat{\mathbb{E}}(L_2) &= \frac{1}{n}\big[a(s-m_{2t})+bl_{2t}\big]. \end{aligned}

The ROC points (specific to the threshold $t$) of models 1 and 2 have coordinates $(l_{1t},m_{1t})$ and $(l_{2t},m_{2t})$, respectively. If the former point dominates the latter point, we have $l_{1t}\leq l_{2t}$ and $m_{1t}\geq m_{2t}$ and at least one of the two inequalities is strict.

What does this imply regarding $\hat{\mathbb{E}}(L_1)$ vs. $\hat{\mathbb{E}}(L_2)$? Since $a,b>0$ and $s-m_{1t},s-m_{2t}\geq0$, then looking at the formulas above we immediately see that $\hat{\mathbb{E}}(L_1)<\hat{\mathbb{E}}(L_2)$. Thus if the ROC point of model 1 dominates the ROC point of model 2, model 1 has lower estimated expected loss than model 2.

*The relevant threshold would be the optimal one. **{ Is above and to the left } OR { is above and not to the right } OR { is to the left and not below }.

Question 2: still open.

Given a threshold* $t$, model 1 has lower estimated expected loss than model 2 if the corresponding ROC point of model 1 dominates** the ROC point of model 2. Here is why.

Let the confusion matrix corresponding to a particular threshold $t$ be $$ \text{Conf}_t=\begin{pmatrix} j_t & k_t\\ l_t & m_t \end{pmatrix} $$ with predicted classes in rows (row 1 ~ class 0, row 2 ~ class 1) and actual classes in columns (column 1 ~ class 0, column 2 ~ class 1). Concretely, $$ \text{Conf}_t=\begin{pmatrix} \#\{{\hat Y=0 \cap Y=0\}}_t & \#\{{\hat Y=0\cap Y=1\}}_t \\ \#\{{\hat Y=1\cap Y=0\}}_t & \#\{{\hat Y=1\cap Y=1\}}_t \end{pmatrix} $$ with $\#$ counting the number of elements that satisfy the condition. We will later add a subscript 1 for model 1 and 2 for model 2.

For any given sample, the number of actual zeros $j_t+l_t$ and the number of actual unities $k_t+m_t$ are fixed at $r$ and $s$, respectively: \begin{aligned} j_t+l_t &= r \quad \text{and} \\ k_t+m_t &= s. \end{aligned} We will make use of the latter equality in a subsequent step. Let us also define the sample size $$ n:=j_t+k_t+l_t+m_t. $$

The estimated expected loss of a model is \begin{aligned} \hat{\mathbb{E}}(L) &= \frac{1}{n}\big[ak_{t}+bl_{t}\big] \\ &= \frac{1}{n}\big[a(s-m_{t})+bl_{t}\big]. \end{aligned} Explicitly, the estimated expected losses of models 1 and 2 are \begin{aligned} \hat{\mathbb{E}}(L_1) &= \frac{1}{n}\big[a(s-m_{1t})+bl_{1t}\big] \quad \text{and} \\ \hat{\mathbb{E}}(L_2) &= \frac{1}{n}\big[a(s-m_{2t})+bl_{2t}\big]. \end{aligned}

The ROC points (specific to the threshold $t$) of models 1 and 2 have coordinates $(l_{1t},m_{1t})$ and $(l_{2t},m_{2t})$, respectively. If the former point dominates the latter point, we have $l_{1t}\leq l_{2t}$ and $m_{1t}\geq m_{2t}$ and at least one of the two inequalities is strict.

What does this imply regarding $\hat{\mathbb{E}}(L_1)$ vs. $\hat{\mathbb{E}}(L_2)$? Since $a,b>0$ and $s-m_{1t},s-m_{2t}\geq0$, then looking at the formulas above we immediately see that $\hat{\mathbb{E}}(L_1)<\hat{\mathbb{E}}(L_2)$. Thus if the ROC point of model 1 dominates the ROC point of model 2, model 1 has lower estimated expected loss than model 2.

*The relevant threshold would be the optimal one. **{ Is above and to the left } OR { is above and not to the right } OR { is to the left and not below }.

Question 1

Given a threshold* $t$, model 1 has lower estimated expected loss than model 2 if the corresponding ROC point of model 1 dominates** the ROC point of model 2. Here is why.

Let the confusion matrix corresponding to a particular threshold $t$ be $$ \text{Conf}_t=\begin{pmatrix} j_t & k_t\\ l_t & m_t \end{pmatrix} $$ with predicted classes in rows (row 1 ~ class 0, row 2 ~ class 1) and actual classes in columns (column 1 ~ class 0, column 2 ~ class 1). Concretely, $$ \text{Conf}_t=\begin{pmatrix} \#\{{\hat Y=0|Y=0\}}_t & \#\{{\hat Y=0|Y=1\}}_t \\ \#\{{\hat Y=1|Y=0\}}_t & \#\{{\hat Y=1|Y=1\}}_t \end{pmatrix} $$ with $\#$ counting the number of elements that satisfy the condition. We will later add a subscript 1 for model 1 and 2 for model 2.

For any given sample, the number of actual zeros $j_t+l_t$ and the number of actual unities $k_t+m_t$ are fixed at $r$ and $s$, respectively: \begin{aligned} j_t+l_t &= r \quad \text{and} \\ k_t+m_t &= s. \end{aligned} We will make use of the latter equality in a subsequent step. Let us also define the sample size $$ n:=j_t+k_t+l_t+m_t. $$

The estimated expected loss of a model is \begin{aligned} \hat{\mathbb{E}}(L) &= \frac{1}{n}\big[ak_{t}+bl_{t}\big] \\ &= \frac{1}{n}\big[a(s-m_{t})+bl_{t}\big]. \end{aligned} Explicitly, the estimated expected losses of models 1 and 2 are \begin{aligned} \hat{\mathbb{E}}(L_1) &= \frac{1}{n}\big[a(s-m_{1t})+bl_{1t}\big] \quad \text{and} \\ \hat{\mathbb{E}}(L_2) &= \frac{1}{n}\big[a(s-m_{2t})+bl_{2t}\big]. \end{aligned}

The ROC points (specific to the threshold $t$) of models 1 and 2 have coordinates $(l_{1t},m_{1t})$ and $(l_{2t},m_{2t})$, respectively. If the former point dominates the latter point, we have $l_{1t}\leq l_{2t}$ and $m_{1t}\geq m_{2t}$ and at least one of the two inequalities is strict.

What does this imply regarding $\hat{\mathbb{E}}(L_1)$ vs. $\hat{\mathbb{E}}(L_2)$? Since $a,b>0$ and $s-m_{1t},s-m_{2t}\geq0$, then looking at the formulas above we immediately see that $\hat{\mathbb{E}}(L_1)<\hat{\mathbb{E}}(L_2)$. Thus if the ROC point of model 1 dominates the ROC point of model 2, model 1 has lower estimated expected loss than model 2.

*The relevant threshold would be the optimal one. **{ Is above and to the left } OR { is above and not to the right } OR { is to the left and not below }.

Question 2: still open.

Question 1

Given a threshold* $t$, model 1 has lower estimated expected loss than model 2 if the corresponding ROC point of model 1 dominates** the ROC point of model 2. Here is why.

Let the confusion matrix corresponding to a particular threshold $t$ be $$ \text{Conf}_t=\begin{pmatrix} j_t & k_t\\ l_t & m_t \end{pmatrix} $$ with predicted classes in rows (row 1 ~ class 0, row 2 ~ class 1) and actual classes in columns (column 1 ~ class 0, column 2 ~ class 1). Concretely, $$ \text{Conf}_t=\begin{pmatrix} \#\{{\hat Y=0 \cap Y=0\}}_t & \#\{{\hat Y=0\cap Y=1\}}_t \\ \#\{{\hat Y=1\cap Y=0\}}_t & \#\{{\hat Y=1\cap Y=1\}}_t \end{pmatrix} $$ with $\#$ counting the number of elements that satisfy the condition. We will later add a subscript 1 for model 1 and 2 for model 2.

For any given sample, the number of actual zeros $j_t+l_t$ and the number of actual unities $k_t+m_t$ are fixed at $r$ and $s$, respectively: \begin{aligned} j_t+l_t &= r \quad \text{and} \\ k_t+m_t &= s. \end{aligned} We will make use of the latter equality in a subsequent step. Let us also define the sample size $$ n:=j_t+k_t+l_t+m_t. $$

The estimated expected loss of a model is \begin{aligned} \hat{\mathbb{E}}(L) &= \frac{1}{n}\big[ak_{t}+bl_{t}\big] \\ &= \frac{1}{n}\big[a(s-m_{t})+bl_{t}\big]. \end{aligned} Explicitly, the estimated expected losses of models 1 and 2 are \begin{aligned} \hat{\mathbb{E}}(L_1) &= \frac{1}{n}\big[a(s-m_{1t})+bl_{1t}\big] \quad \text{and} \\ \hat{\mathbb{E}}(L_2) &= \frac{1}{n}\big[a(s-m_{2t})+bl_{2t}\big]. \end{aligned}

The ROC points (specific to the threshold $t$) of models 1 and 2 have coordinates $(l_{1t},m_{1t})$ and $(l_{2t},m_{2t})$, respectively. If the former point dominates the latter point, we have $l_{1t}\leq l_{2t}$ and $m_{1t}\geq m_{2t}$ and at least one of the two inequalities is strict.

What does this imply regarding $\hat{\mathbb{E}}(L_1)$ vs. $\hat{\mathbb{E}}(L_2)$? Since $a,b>0$ and $s-m_{1t},s-m_{2t}\geq0$, then looking at the formulas above we immediately see that $\hat{\mathbb{E}}(L_1)<\hat{\mathbb{E}}(L_2)$. Thus if the ROC point of model 1 dominates the ROC point of model 2, model 1 has lower estimated expected loss than model 2.

*The relevant threshold would be the optimal one. **{ Is above and to the left } OR { is above and not to the right } OR { is to the left and not below }.

Question 2: still open.

Question 1

Given a threshold* $t$ (e.g. the optimal one), model 1 has lower estimated expected loss than model 2 if the corresponding ROC point of model 1 dominates** the ROC point of model 2. Here is why.

Let the confusion matrix corresponding to a particular threshold $t$ be $\text{Conf}_t=\begin{pmatrix} j_t & k_t\\ l_t & m_t \end{pmatrix}$ with predicted classes in rows (row 1 ~ class 0, row 2 ~ class 1) and actual classes in columns (column 1 ~ class 0, column 2 ~ class 1). Concretely, $\text{Conf}_t=\begin{pmatrix} \#\{{\hat Y=0|Y=0\}}_t & \#\{{\hat Y=0|Y=1\}}_t \\ \#\{{\hat Y=1|Y=0\}}_t & \#\{{\hat Y=1|Y=1\}}_t \end{pmatrix}$ with $\#$ counting the number of elements that satisfy the condition. We will later add a subscript 1 for model 1 and 2 for model 2.

For any given sample, the number of actual zeros $j_t+l_t$ and the number of actual unities $k_t+m_t$ are fixed at $r$ and $s$, respectively: $j_t+l_t=r$ and $k_t+m_t=s$. We will make use of the latter equality in a subsequent step. Let us also define the sample size $n:=j_t+k_t+l_t+m_t$.

The estimated expected loss of a model is $\hat{\mathbb{E}}(L)=\frac{1}{n}\big[ak_{t}+bl_{t}\big]$. Explicitly, the estimated expected losses of models 1 and 2 are $\hat{\mathbb{E}}(L_1)=\frac{1}{n}\big[ak_{1t}+bl_{1t}\big]$ and $\hat{\mathbb{E}}(L_2)=\frac{1}{n}\big[ak_{2t}+bl_{2t}\big]$, respectively.

The ROC points (specific to the threshold $t$) of models 1 and 2 have coordinates $(l_{1t},m_{1t})$ and $(l_{2t},m_{2t})$, respectively. If the former point dominates the latter point, we have $l_{1t}\leq l_{2t}$ and $m_{1t}\geq m_{2t}$ and at least one of the two inequalities is strict. What does this imply regarding $\hat{\mathbb{E}}(L_1)$ vs. $\hat{\mathbb{E}}(L_2)$?

$\hat{\mathbb{E}}(L_1)=\frac{1}{n}\big[ak_{1t}+bl_{1t}\big]=\frac{1}{n}\big[a(s-m_{1t})+bl_{1t}\big]$,
$\hat{\mathbb{E}}(L_2)=\frac{1}{n}\big[ak_{2t}+bl_{2t}\big]=\frac{1}{n}\big[a(s-m_{2t})+bl_{2t}\big]$.

Since $a,b>0$ and $s-m_{1t},s-m_{2t}\geq0$, then obviously $\hat{\mathbb{E}}(L_1)<\hat{\mathbb{E}}(L_2)$.

Thus if the ROC point of model 1 dominates the ROC point of model 2, model 1 has lower estimated expected loss than model 2.

*{ Is above and to the left } OR { is above and not to the right } OR { is to the left and not below }.
**The relevant pair of points would be the one corresponding to the optimal threshold.

Question 2: still open.

Question 1

Given a threshold* $t$, model 1 has lower estimated expected loss than model 2 if the corresponding ROC point of model 1 dominates** the ROC point of model 2. Here is why.

Let the confusion matrix corresponding to a particular threshold $t$ be $$ \text{Conf}_t=\begin{pmatrix} j_t & k_t\\ l_t & m_t \end{pmatrix} $$ with predicted classes in rows (row 1 ~ class 0, row 2 ~ class 1) and actual classes in columns (column 1 ~ class 0, column 2 ~ class 1). Concretely, $$ \text{Conf}_t=\begin{pmatrix} \#\{{\hat Y=0|Y=0\}}_t & \#\{{\hat Y=0|Y=1\}}_t \\ \#\{{\hat Y=1|Y=0\}}_t & \#\{{\hat Y=1|Y=1\}}_t \end{pmatrix} $$ with $\#$ counting the number of elements that satisfy the condition. We will later add a subscript 1 for model 1 and 2 for model 2.

For any given sample, the number of actual zeros $j_t+l_t$ and the number of actual unities $k_t+m_t$ are fixed at $r$ and $s$, respectively: \begin{aligned} j_t+l_t &= r \quad \text{and} \\ k_t+m_t &= s. \end{aligned} We will make use of the latter equality in a subsequent step. Let us also define the sample size $$ n:=j_t+k_t+l_t+m_t. $$

The estimated expected loss of a model is \begin{aligned} \hat{\mathbb{E}}(L) &= \frac{1}{n}\big[ak_{t}+bl_{t}\big] \\ &= \frac{1}{n}\big[a(s-m_{t})+bl_{t}\big]. \end{aligned} Explicitly, the estimated expected losses of models 1 and 2 are \begin{aligned} \hat{\mathbb{E}}(L_1) &= \frac{1}{n}\big[a(s-m_{1t})+bl_{1t}\big] \quad \text{and} \\ \hat{\mathbb{E}}(L_2) &= \frac{1}{n}\big[a(s-m_{2t})+bl_{2t}\big]. \end{aligned}

The ROC points (specific to the threshold $t$) of models 1 and 2 have coordinates $(l_{1t},m_{1t})$ and $(l_{2t},m_{2t})$, respectively. If the former point dominates the latter point, we have $l_{1t}\leq l_{2t}$ and $m_{1t}\geq m_{2t}$ and at least one of the two inequalities is strict. 

What does this imply regarding $\hat{\mathbb{E}}(L_1)$ vs. $\hat{\mathbb{E}}(L_2)$? Since $a,b>0$ and $s-m_{1t},s-m_{2t}\geq0$, then looking at the formulas above we immediately see that $\hat{\mathbb{E}}(L_1)<\hat{\mathbb{E}}(L_2)$. Thus if the ROC point of model 1 dominates the ROC point of model 2, model 1 has lower estimated expected loss than model 2.

*The relevant threshold would be the optimal one. **{ Is above and to the left } OR { is above and not to the right } OR { is to the left and not below }.

Question 2: still open.