Which performance metrics for highly imbalanced multiclass dataset?

Question

I have a dataset with 5 classes. About 98% of the dataset belong to class 5. Classes 1-4 share equally about 2% of the dataset. However, it is highly important, that classes 1-4 are correctly classified.

The accuracy is not a good measure of performance for my task. I found lots of information on metrics for imbalanced binary classification tasks but not on multiclass problems.

Which performance metrics should I use for such a task?

• TP, TN, FP, FN
• Precision
• Sensitivity
• Specificity
• F-score
• ROC-AUC (micro, macro, samples, weighted)

Answer 1

For unbalanced classes, I would suggest to go with Weighted F1-Score or Average AUC/Weighted AUC

Let's first see F1-Score for binary classification.

The F1-score gives a larger weight to lower numbers.

For example,

• when Precision is 100% and Recall is 0%, the F1-score will be 0%, not 50%.
• When let us say, we have Classifier A with precision=recall=80%, and Classifier B has precision=60%, recall=100%. Arithmetically, the mean of the precision and recall is the same for both models. But when we use F1’s harmonic mean formula, the score for Classifier A will be 80%, and for Classifier B it will be only 75%. Model B’s low precision score pulled down its F1-score.

Now, come to the Mutliclass Classification

Let us suppose we have the five classes, class_1, class_2, class_3, class_4, class_5

and the model is having the following results for each class.

Formula for precision for each class = (True Positive for class)/(Count of predicted Positive for that class)

e.g. precision for class_1 = (True Positive for class_1)/(Count of Predicted of class_1)

Formula for Recall for each class = (True Positive for class)/(Actual Positive for that class)

e.g. precision for class_1 = (True Positive for class_1)/(Total instances of class_1)

Formula for F1: F1 is the geometric mean of Precision and Recall i.e.

F1 = 2*(Precision*Recall)/(Precision+Recall)

Macro-F1 = Average(Class_1_F1 + Class_2_F1 + Class_3_F1 + Class_4_F1 + Class_5_F1)

Macro-Precision = Average(Class_1_Precision + Class_2_Precision + Class_3_Precision + Class_4_Precision + Class_5_Precision)

Macro-Recall = Average(Class_1_Recall + Class_2_Recall + Class_3_Recall + Class_4_Recall + Class_5_Recall)

Problem with Macro calculation: When averaging the macro-F1, we gave equal weights to each class.

Weighted F1 Score:

We don’t have to do that: in weighted-average F1-score, or weighted-F1, we weight the F1-score of each class by the number of samples from that class.

Weighted F1 Score = (N1*Class_1_F1 + N2*Class_2_F1 + N3*Class_3_F1 + N4*Class_4_F1 + N5*Class_5_F1)/(N1 + N2 + N3 + N4 + N5)

References: https://towardsdatascience.com/multi-class-metrics-made-simple-part-ii-the-f1-score-ebe8b2c2ca1

Answer 2

Precision, recall, F1, ROC/AUC, and other metrics like specificity/sensitivity that you mentioned can be good for multi-class imbalanced metrics. If you want to emphasize the undersampled classes, use macro weighting (arithmetic average). If not, use micro average, which is weighted by number of samples.

Another metric I don't see people often talking about is Cohen's Kappa. I like to think of it like accuracy, but taking into account the "no information rate" or random guessing baseline. It will give you a score similar to accuracy, though I believe it has some flaws in certain situations. In general, I've found Cohen's Kappa to work well.

Others include the litany of metrics listed on Wikipedia's confusion matrix page, such as Matthew's correlation coefficient (MCC) and others.