1
$\begingroup$

I have an unbalanced multi-class classification problem with the following class distributions:

Class 0: 17.1% 
Class 1: 63.2% 
Class 2: 19.7%

I am using scikit-learn's Support Vector Classifier with 'balanced' class weights to classify samples into one of the three classes. However, I don't know what would be the most suitable performance metric to evaluate the result? So far I have been using F1-micro score, but not sure if this is the best option for multi-class imbalance problems?

$\endgroup$
1
$\begingroup$

There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, DP, etc.

Disclaimer:

If you use python, PyCM module can help you to find and calculate these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})  

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

>>> score = cm.Kappa
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.