Heuristic argument for matching a confusion matrix with a cost matrix

Question

Logistic regression was applied on a dataset. It produced fitted class probabilities.
Class labels were assigned the first time using a threshold determined from cost matrix no. 1. (The threshold is selected to minimize the total cost.) A corresponding confusion matrix no. 1 was obtained.
Class labels were assigned the second time using a threshold determined from cost matrix no. 2. A corresponding confusion matrix no. 2 was obtained.

Exercise: Match the confusion matrices to the corresponding cost matrices.

The two confusion matrices are $$ A = \begin{pmatrix} 88 & 200 \\ 2 & 58 \end{pmatrix} \quad \text{and} \quad B = \begin{pmatrix} 76 & 126 \\ 14 & 132 \end{pmatrix}. $$
The two cost matrices are $$ C = \begin{pmatrix} 0 & 11 \\ 55 & 0 \end{pmatrix} \quad \text{and} \quad D = \begin{pmatrix} 0 & 6 \\ 60 & 0 \end{pmatrix}. $$

Solution: Two heuristic solutions have been proposed.

• Heuristic solution 1: Both cost matrices penalize predicting class 2 and being wrong about that more heavily than predicting class 1 and being wrong about that. The difference is that in $D$ this is more pronounced than in $C$. Thus we would expect to see fewer instances in the bottom left corner of the confusion matrix corresponding to $D$ than to $C$. Therefore, $A$ ($2$ instances) matches $D$ and $B$ ($14$ instances) matches $C$.

• Heuristic solution 2: Let us obtain the total cost of the two possible matches and see which one is lower. That will indicate the correct match. Match 1 $(A,C)$ and $(B,D)$ yields a total cost of $3906$. Match 2 $(A,D)$ and $(B,C)$ yields a total cost of $3476$. Therefore, Match 2 is the correct match.

Question: Does the heuristic solution 2 make sense? If not, construct a counterexample to show its lack of validity.

This is not self study. It came up while grading homework solutions.

Answer 1

I think heuristic solution 2 does make sense.

We know that both $A$ and $B$ are actual confusion matrices obtained from the dataset using different classification thresholds. Thus, by varying the classification threshold one may produce both $A$ and $B$.

Consider the cost matrix $C$. It produced a confusion matrix that is either $A$ or $B$. The threshold was selected so as to minimize the total cost. The cost due to $A$ under $C$ is $2310$; the cost due to $B$ under $C$ is $2156$. Both $A$ and $B$ are feasible (see the previous paragraph) and the latter produces a lower total cost. If anything, $C$ cannot have yielded $A$ because a better solution is possible (feasible), and it is $B$.

Analogous argument suggests that $D$ cannot have yielded $B$ (total cost $1596$) as $A$ (total cost $1320$) yields a lower total cost and is feasible.

From the previous two paragraphs it follows that the total cost of both pairs resulting from the correct match must be lower than the total cost of both pairs resulting from the wrong match. Thus you can use the total cost of each possible match as a basis for finding out the correct match (it will be the one minimizing the total cost).