Evaluating multiclass imbalanced problem per class

Question

For a multiclass imbalanced problem, accuracy is not a good metric to evaluate model performance. Equally, accuracy is a global metric, so nothing like accuracy per-class (doesn't make sense).

Scikit-learn provides the classification_report function so one can evaluate model's precision/recall per class, e.g:

classification_report(y_true, y_pred, target_names=target_names)
              precision    recall  f1-score   support

     Class:0      0.703     0.896     0.788      4491
     Class:1      0.048     0.147     0.072        75
     Class:2      0.368     0.503     0.425      1097
     Class:3      0.937     0.850     0.892     17162
     Class:4      0.529     0.177     0.265       311

    accuracy                          0.832     23136
   macro avg      0.517     0.515     0.488     23136
weighted avg      0.856     0.832     0.838     23136

Are there other metrics that evaluate per-class so I can evaluate my model across more metrics than precision/recall/f1? The goal is to assess the model on a per-class basis.

Answer 1

Precision and recall are misleading just like accuracy is. Every criticism against accuracy at Why is accuracy not the best measure for assessing classification models? applies equally to precision and recall.

Use probabilistic predictions for class membership, and evaluate these using proper scoring-rules, e.g., the Brier score or the log score. More information and pointers to literature can be found at the tag wiki.

If you really want to, you can calculate the average of such scores over instances that fall into particular classes, but the properness of these scores refers to their overall performance, and I suspect that looking at scores per class may be misleading.

You may be interested in this earlier answer of mine.