Metrics for unbalanced classes [duplicate]

Question

I have been looking for good metrics on this data set I am working, however it is highly unbalanced. It has a total of 8 categorical classes, one of them is responsible for ~40% of data, another for 20% and every other has around the same quantity. I was able to create a model reaching 94% accuracy and another reached 91% accuracy, however when I check the confusion matrix of the second it hits better in the classes with fewer examples.

I know for this case accuracy is not the best metric, however every post/paper I read (like this and this) just evaluate two classes. My question is: what other metrics, beside accuracy, should I use? What score tells me what model is the best for this poroblem?

Regards!

Answer 1

To answer your question, some metrics used in such situations:

• F1-score.
• Kappa.
• Geometric Mean Score
• Intersection Over Union (Jaccard)

All of which essentially weigh the confusion matrix elements in a different way.

---

Aggregation

Besides selecting which metric suits your task the best, you should also consider isn't what metric you should use but how to aggregate the results from each class.

If you want you objectively evaluate the performance of your model and have each class matter the same regardless of the number of samples in the test set you could just macro-average the results!

Macro-averaging simply averages the metric's scores for each classes. This doesn't take into account at all class imbalance, regardless of the metric you choose.

Example:
(I'm going to use accuracy to prove my point. Generally speaking is a bad metric for class imbalance and you should avoid it.):

Class 1: $135/150$ (correct/total) --> $90\%$ (accuracy for class 1)
Class 2: $15/30$ --> $50\%$
Class 3: $5/20$ --> $20\%$

micro-averaged accuracy: $\frac{135 + 15 + 5}{150 + 30 + 20} = \frac{155}{200} = 77.5\%$ which is a very high score, not indicative of the model's actual performance because of the high imbalance.
macro-averaged accuracy: $\frac{90 + 50 + 20}{3}\% = 53.3\%$

Macro-averaging does have its issues, mainly that each sample of an under-represented class essentially counts more to the score than each one from the majority class. So in the example above if we made $10$ more correct predictions for class 3 then it would boost the overall performance of the model from $53.3\%$ to $71.6\%$, which is a massive improvement, that might simple occure due to a high bias towards class 3.

So see if it's right for you and use with caution. All metrics are good as long as you are aware what they represent.

Answer 2

You should definetly use the F1 score which takes into account the mistakes on positives and negatives for all classes. I deliver you to this article: https://towardsdatascience.com/accuracy-precision-recall-or-f1-331fb37c5cb9