13
$\begingroup$

I have a dataset with 5 classes. About 98% of the dataset belong to class 5. Classes 1-4 share equally about 2% of the dataset. However, it is highly important, that classes 1-4 are correctly classified.

The accuracy is not a good measure of performance for my task. I found lots of information on metrics for imbalanced binary classification tasks but not on multiclass problems.

Which performance metrics should I use for such a task?

  • TP, TN, FP, FN
  • Precision
  • Sensitivity
  • Specificity
  • F-score
  • ROC-AUC (micro, macro, samples, weighted)
$\endgroup$
1

2 Answers 2

7
$\begingroup$

For unbalanced classes, I would suggest to go with Weighted F1-Score or Average AUC/Weighted AUC

Let's first see F1-Score for binary classification.

The F1-score gives a larger weight to lower numbers.

For example,

  • when Precision is 100% and Recall is 0%, the F1-score will be 0%, not 50%.
  • When let us say, we have Classifier A with precision=recall=80%, and Classifier B has precision=60%, recall=100%. Arithmetically, the mean of the precision and recall is the same for both models. But when we use F1’s harmonic mean formula, the score for Classifier A will be 80%, and for Classifier B it will be only 75%. Model B’s low precision score pulled down its F1-score.

Now, come to the Mutliclass Classification

Let us suppose we have the five classes, class_1, class_2, class_3, class_4, class_5

and the model is having the following results for each class.

enter image description here

Formula for precision for each class = (True Positive for class)/(Count of predicted Positive for that class)

e.g. precision for class_1 = (True Positive for class_1)/(Count of Predicted of class_1)

Formula for Recall for each class = (True Positive for class)/(Actual Positive for that class)

e.g. precision for class_1 = (True Positive for class_1)/(Total instances of class_1)

Formula for F1: F1 is the geometric mean of Precision and Recall i.e.

F1 = 2*(Precision*Recall)/(Precision+Recall)

Macro-F1 = Average(Class_1_F1 + Class_2_F1 + Class_3_F1 + Class_4_F1 + Class_5_F1)

Macro-Precision = Average(Class_1_Precision + Class_2_Precision + Class_3_Precision + Class_4_Precision + Class_5_Precision)

Macro-Recall = Average(Class_1_Recall + Class_2_Recall + Class_3_Recall + Class_4_Recall + Class_5_Recall)

Problem with Macro calculation: When averaging the macro-F1, we gave equal weights to each class.

Weighted F1 Score:

We don’t have to do that: in weighted-average F1-score, or weighted-F1, we weight the F1-score of each class by the number of samples from that class.

Weighted F1 Score = (N1*Class_1_F1 + N2*Class_2_F1 + N3*Class_3_F1 + N4*Class_4_F1 + N5*Class_5_F1)/(N1 + N2 + N3 + N4 + N5)

References: https://towardsdatascience.com/multi-class-metrics-made-simple-part-ii-the-f1-score-ebe8b2c2ca1

$\endgroup$
3
  • 5
    $\begingroup$ Just for the record: Medium articles are not credible sources. (I am not down-voting this post or anything like that, but I have seen "bad" Medium articles from experienced users; so be careful about them.) $\endgroup$
    – usεr11852
    Apr 28, 2020 at 14:38
  • 2
    $\begingroup$ Thanks for that, i checked the profile of the writer, he is a P.hD student and have good experience with Data Science. Apart from this, I am a Data Scientist, I have used this Metric for our models evaluation. $\endgroup$ Apr 28, 2020 at 15:23
  • 1
    $\begingroup$ Why is giving equal weights to each class in Macro F1 a bad thing? For example, Let us assume that class one is the majority class, and Class_1_F1 is the highest as well. wouldn't this mean Weighted F1 is represents F1 of the majority class 1? But we want a metric that represents the minority classes, isn't it? $\endgroup$
    – Minh Khôi
    Mar 27, 2021 at 6:50
0
$\begingroup$

Precision, recall, F1, ROC/AUC, and other metrics like specificity/sensitivity that you mentioned can be good for multi-class imbalanced metrics. If you want to emphasize the undersampled classes, use macro weighting (arithmetic average). If not, use micro average, which is weighted by number of samples.

Another metric I don't see people often talking about is Cohen's Kappa. I like to think of it like accuracy, but taking into account the "no information rate" or random guessing baseline. It will give you a score similar to accuracy, though I believe it has some flaws in certain situations. In general, I've found Cohen's Kappa to work well.

Others include the litany of metrics listed on Wikipedia's confusion matrix page, such as Matthew's correlation coefficient (MCC) and others.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.