How can I compute precision, recall and F-measure when we define a cost matrix for evaluation?
In my problem I have 3 different classes, and some errors are more dangerous than others. I would like my evaluation metrics to take into account this information.
In my problem I have 3 different classes, and some errors are more dangerous than others. I would like my evaluation metrics to take into account this information.
Misclassification cost should cover this sufficiently in my opinion.
Assuming a classification problem with 3 classes $$Y=\{y_1,y_2,y_3\}$$ and a cost matrix $$C=\begin{bmatrix}0 & c_{12} & c_{13}\\ c_{21} & 0 & c_{23}\\c_{31}&c_{32}&0 \end{bmatrix}$$
with the diagonal indicating no costs for correct classification and cost $c_{12}$ entailed by predicting an instance of $y_1$ to be $y_2$ instead. Not that the cost matrix need not by symmetric and, based on your problem description, shouldn't be.
From your model, generate the confusion matrix (I will denote it as $\hat{Y}$ for lack of other notation) for your validation set and multiply it with $C$ to get the misclassification cost of your model
$$ MCC= \hat{Y} \times C $$
When evaluating different models, lower misclassification cost is of course preferred
Note that, the term precision and recall is defined in binary classification case. (Same as other terms, such as false positive, false negative, etc.)
If you have 3 classes and want to use the precision and recall metrics, you need to define what is "positive" or "negative". One vs. all others is one way of defining it. For example, I want to classify three animals, cat dog and rabbit, I can define detecting cat is "positive". After the definition, these metrics (precision recall, false positive rate etc.) can be used.
A related discussion can be found in my answer here
