Skip to main content
1 of 2
Richard Hardy
  • 69.5k
  • 13
  • 126
  • 278

Evaluating classification results when importance of correct classification varies with class

Suppose we have a categorical variable $Y$ and we are trying to classify it. Our decision (the predicted class for $Y$) is $\hat Y$. We are facing a loss function which can be represented by a loss matrix $L$ (a matrix with losses in each cell) with a typical element $l_{ij}$. They layout of $L$ corresponds to the confusion matrix where rows correspond to actual classes and columns to predicted ones. Each off-diagonal element $l_{ij}$ ($i\neq j$) is the loss associated with the specific misclassification. In the case where $Y$ must belong to one of two categories, $L$ can look something like $$ L=\left(\begin{array}{ccc}0 & 3\\2 & 0\end{array}\right). $$ Consider a situation where correctly predicting that $Y$ belongs to the first category is more important than correctly predicting that $Y$ belongs to the second category. E.g. it can be more important to (correctly) classify a malignant tumor as malignant (so that we can take action immediately) than to (correctly) classify a benign one as benigm. That would suggest the diagonal elements of $L$ need not all be equal. E.g., we could have
$$ L=\left(\begin{array}{ccc}0 & 3\\2 & 1\end{array}\right). $$ Does that make sense?
Is it common practice to have nonequal diagonal elements in $L$?

Richard Hardy
  • 69.5k
  • 13
  • 126
  • 278