# Evaluating classification results when importance of correct classification varies with class

Suppose we have a categorical variable $$Y$$ and we are trying to classify it. Our decision (the predicted class for $$Y$$) is $$\hat Y$$. We are facing a loss function which can be represented by a loss matrix $$L$$ (a matrix with losses in each cell) with a typical element $$l_{ij}$$. They layout of $$L$$ corresponds to the confusion matrix where rows correspond to actual classes and columns to predicted ones. Each off-diagonal element $$l_{ij}$$ ($$i\neq j$$) is the loss associated with the specific misclassification. In the case where $$Y$$ must belong to one of two categories, $$L$$ can look something like $$L=\left(\begin{array}{ccc}0 & 3\\2 & 0\end{array}\right).$$ Consider a situation where correctly predicting that $$Y$$ belongs to the first category is more important than correctly predicting that $$Y$$ belongs to the second category. E.g. it can be more important to (correctly) classify a malignant tumor as malignant (so that we can take action immediately) than to (correctly) classify a benign one as benigm. That would suggest the diagonal elements of $$L$$ need not all be equal. E.g., we could have
$$L=\left(\begin{array}{ccc}0 & 3\\2 & 1\end{array}\right).$$ (The diagonal element should of course be the smallest in its own row, otherwise there would be incentives for misclassification.)

Does that make sense?
Is it common practice to have nonequal diagonal elements in $$L$$?

• I wonder how this would work with a proper scoring rule instead of extending 0-1 loss, too.
– Dave
Commented Jun 21, 2020 at 15:39