How to fully evaluate a multiclass classification problem?

Question

When you have a multiclass classification problem, what is the right way to evaluate it's performance?

What I usually do is to display the confusion matrix and the classification_report() offered by the scikit-learn python library.

However I wonder why nobody ever calculates the Precision vs. Recall and the ROC curves. Should they be calculated as well?

I found the following example which calculates them but when I try to reverse-engineer the problem to calculate for example the precision and recall of each class I do not get the same results as the ones from the classification_report() (as you can see here).

QUESTION: Would you be able to provide an example to have a full analysis of the performance of a multiclass classification problem?

Answer 1

Precision and recall are not straightforward to define and calculate in the multiclass situation, and I would argue that they can be badly misleading even in the two-class situation, since every criticism against accuracy applies equally to precision and recall. (People do disagree with me here.)

The generalization to the ROC to multiple classes is even less clear than for precision and recall.

I would always use a model that gives me predicted class membership probabilities, not hard 0-1 classifications. There is a difference between predicted probabilities of 0.98-0.01-0.01 and 0.4-0.3-0.3, even if the most likely class is the first one in both cases.

Probabilistic predictions can be evaluated using proper scoring rules. Two very common proper scoring rules that can be used in multiclass situations are the Brier and the log score.