7
$\begingroup$

I am trying to understand the evaluation metrics for a classifier model.

What is the necessity for finding out Hamming Loss ?

I have read some documents on the Internet, which basically relates Hamming Loss to a Multi-classifier but still couldn't really understand why it is really needed to evaluate the model.

Also, is Hamming Loss actually just 1-Accuracy for an Imbalanced Binary Classifier ?

What does it bring to the table that Precision, Recall, F1-Measure couldn't ?

$\endgroup$
1
  • 1
    $\begingroup$ I think a main question is if Hamming Loss is relevant for an imbalanced classification task. From what I understand Hamming Loss is mostly relevant to Multi-label classification and not Multi-class classification. In that respect associating with a binary classification task seems unnecessary. $\endgroup$
    – usεr11852
    Apr 13, 2018 at 20:24

3 Answers 3

4
$\begingroup$

The hamming loss (HL) is

the fraction of the wrong labels to the total number of labels

Hence, for the binary case (imbalanced or not), HL=1-Accuracy as you wrote.

When considering the multi label use case, you should decide how to extend accuracy to this case. The method choose in hamming loss was to give each label equal weight. One could use other methods (e.g., taking the maximum).

Since hamming loss is designed for multi class while Precision, Recall, F1-Measure are designed for the binary class, it is better to compare the last one to Accuracy. In general, there is no magical metric that is the best for every problem. In every problem you have different needs, and you should optimize for them.

By the way, specifically for imbalanced problems, accuracy is a problematic metric. For details, see here.

$\endgroup$
6
+50
$\begingroup$

Since Hamming loss is defined as $$HL = \frac{1}{N L} \sum_{l=1}^L\sum_{i=1}^N Y_{i,l} \oplus X_{i,l},$$ where $\oplus$ denotes exlusive-or, $X_{i,l}$ ($Y_{i,l}$) stands for boolean that the $i$-th datum (prediction) contains the $l$-th label, it really equals to (1 - accuracy) for binary case $(L=1)$: $$HL=\frac{1}{N}\sum_{i=1}^N Y_i \oplus X_i = \frac{1}{N}\sum_{i=1}^N 1 - I(X_i,Y_i) = 1 - \frac{\sum_{i=1}^N I(X_i,Y_i)}{N} =1 - Ac,$$ where $I(X_i, Y_i) = 1$ if $X_i = Y_i$ and 0 otherwise and Ac denotes accuracy.

From the above reason, the use of HL does not make sense to me in the binary case, respectively it is directly related to accuracy. Nevertheless, as mentioned here, the accuracy is ambiguous in the multiple-label case.

The HL thus presents one clear single-performance-value for multiple-label case in contrast to the precision/recall/f1 that can be evaluated only for independent binary classifiers for each label.

$\endgroup$
6
$\begingroup$

In multi-label classification, a misclassification is no longer a hard wrong or right. A prediction containing a subset of the actual classes should be considered better than a prediction that contains none of them.source

So accuracy counts no of correctly classified data instance, Hamming Loss calculates loss generated in the bit string of class labels during prediction, It does that by exclusive or (XOR) between the actual and predicted labels and then average across the dataset. source

  • Number of Instances = 2
  • Number of Labels = 2

Case 1: Actual Same as Predicted

Actual = [[0 1]         Predicted= [[0 1]
          [1 1]]                    [1 1]]

Actual XOR Predicted = [[0 0
                         0 0]]

from sklearn.metrics import hamming_loss
import numpy as np
print(hamming_loss(np.array([[0,1], [1,1]]), np.array([[0,1], [1,1]])))

HL= 0.0

Case 2: Actual completely different from Predicted

Actual = [[0 1]         Predicted= [[1 0]
          [1 1]]                    [0 0]]

Actual XOR Predicted = [[1 1
                         1 1]]

from sklearn.metrics import hamming_loss
import numpy as np
print('HL=',hamming_loss(np.array([[0,1], [1,1]]), np.array([[1,0], [0,0]])))

HL = 4/(2*2) = 1

Case 3: Actual partially different from Predicted

Actual = [[0 1]         Predicted= [[0 0]
          [1 1]]                    [0 1]]

Actual XOR Predicted = [[0 1
                         1 0]]

from sklearn.metrics import hamming_loss
import numpy as np
print(hamming_loss(np.array([[0,1], [1,1]]), np.array([[0,0], [0,1]])))

HL = (1+1)/(2*2) = 0.5

hamming loss value ranges from 0 to 1. Lesser value of hamming loss indicates a better classifier.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.