# Heuristic argument for matching a confusion matrix with a cost matrix

Logistic regression was applied on a dataset. It produced fitted class probabilities.
Class labels were assigned the first time using a threshold determined from cost matrix no. 1. (The threshold is selected to minimize the total cost.) A corresponding confusion matrix no. 1 was obtained.
Class labels were assigned the second time using a threshold determined from cost matrix no. 2. A corresponding confusion matrix no. 2 was obtained.

Exercise: Match the confusion matrices to the corresponding cost matrices.

The two confusion matrices are $$A = \begin{pmatrix} 88 & 200 \\ 2 & 58 \end{pmatrix} \quad \text{and} \quad B = \begin{pmatrix} 76 & 126 \\ 14 & 132 \end{pmatrix}.$$
The two cost matrices are $$C = \begin{pmatrix} 0 & 11 \\ 55 & 0 \end{pmatrix} \quad \text{and} \quad D = \begin{pmatrix} 0 & 6 \\ 60 & 0 \end{pmatrix}.$$

Solution: Two heuristic solutions have been proposed.

• Heuristic solution 1: Both cost matrices penalize predicting class 2 and being wrong about that more heavily than predicting class 1 and being wrong about that. The difference is that in $$D$$ this is more pronounced than in $$C$$. Thus we would expect to see fewer instances in the bottom left corner of the confusion matrix corresponding to $$D$$ than to $$C$$. Therefore, $$A$$ ($$2$$ instances) matches $$D$$ and $$B$$ ($$14$$ instances) matches $$C$$.

• Heuristic solution 2: Let us obtain the total cost of the two possible matches and see which one is lower. That will indicate the correct match. Match 1 $$(A,C)$$ and $$(B,D)$$ yields a total cost of $$3906$$. Match 2 $$(A,D)$$ and $$(B,C)$$ yields a total cost of $$3476$$. Therefore, Match 2 is the correct match.

Question: Does the heuristic solution 2 make sense? If not, construct a counterexample to show its lack of validity.

This is not self study. It came up while grading homework solutions.

We know that both $$A$$ and $$B$$ are actual confusion matrices obtained from the dataset using different classification thresholds. Thus, by varying the classification threshold one may produce both $$A$$ and $$B$$.
Consider the cost matrix $$C$$. It produced a confusion matrix that is either $$A$$ or $$B$$. The threshold was selected so as to minimize the total cost. The cost due to $$A$$ under $$C$$ is $$2310$$; the cost due to $$B$$ under $$C$$ is $$2156$$. Both $$A$$ and $$B$$ are feasible (see the previous paragraph) and the latter produces a lower total cost. If anything, $$C$$ cannot have yielded $$A$$ because a better solution is possible (feasible), and it is $$B$$.
Analogous argument suggests that $$D$$ cannot have yielded $$B$$ (total cost $$1596$$) as $$A$$ (total cost $$1320$$) yields a lower total cost and is feasible.