I have a dataset with unbalanced classes. Three classes make up about 60% of the data. Also, I have different test splits that cause an imbalance. For e.g:
Train set: label_1 ... label_n
Test set: label_1, label_3, label_9
This means that even though I have only 3 labels in my test set, it could potentially be predicted as 1 of n labels. So when I use sklearn.metrics.precision_recall_fscore_support, I get a matrix with a lot of zeros.
My problem is that I need to get an average F-score across all classes, rather than a per-class value. However, just taking an average of the matrix returned from the above sklearn function will always be a very low value since there are so many zeros. On the other hand, taking an average over non-zero values does not make sense to me either since the total number of potential predictions should be the total number of classes.
Is there a good way to take an average in this case? I've tried using the micro, macro and weighted average options but I am not sure which one is right.
Could anyone please help me with this?