I'm working on a multi-label classification problem where I want to classify text into 20 categories, and each text may belong to one or multiple categories. Each category is a binary value of 0 or 1, and is highly imbalanced, i.e. the vast majority are 0 and only small portion are 1. I read about Hamming Loss as a common measure for multi-label classifier, so consider below example:
y_pred = np.array([[0,0,0,0,1,0,0,0,0,0,0,0,1,0,0,0,0,0,0,1],[0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,1],[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]]) y_true = np.array([[0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0],[0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,1],[1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]]) from sklearn.metrics import hamming_loss print(hamming_loss(y_true, y_pred))
This would give a Hamming Loss of 0.083, but since the majority are 0, a random guess of all labels as 0 would also give similar performance:
y_pred2 = np.zeros(y_pred.shape) print(hamming_loss(y_true, y_pred2)) # hamming loss = 0.083
So I'm just wondering for such multi-label classification problem with large number of labels and majority are 0, what would be a proper measure of classifier performance?