In a multi-label classification, the accuracy is commonly defined as [1] $$ \text{Accuracy}(\boldsymbol{Y},\, \boldsymbol{Z}) := \frac{1}{n} \sum_{i=1}^n \frac{\lvert Y_i \cap Z_i \rvert}{\lvert Y_i \cup Z_i \rvert}, $$

where $Y_i$ is the set of predicted labels for sample $n$ and $Z_i$ is the corresponding set of ground-truth labels. If I get things right, this means that we divide the number of correctly classified labels per sample by the number of labels that are either predicted or given as ground-truth per sample and average over. For example, if $\boldsymbol{Y} := ([0,0,1],\,[1,0,1])$ and $\boldsymbol{Z} := ([0,0,0],\, [1,0,1])$, this would give $$ \text{Accuracy}(\boldsymbol{Y},\, \boldsymbol{Z}) = \frac{1}{2}\left( \frac{0}{1} + \frac{2}{2} \right) = \frac{1}{2}. $$

To me, this is unintuitive, since the classwise accuracies are $1$, $1$ and $\frac{1}{2}$ (in that order). The problem is that the definition of multi-label accuracy does not reflect correctly classified absence of labels, whereas the classwise accuracy does. The same is true for multi-label recall and precision as defined in [1].

Is it really like that or do I get the definitions wrong? Can you recommend any multi-label measures that suit better for problems, where correctly classified absence of labels is equally important as presence (apart from the Hamming Loss)?

[1] Sorower, Mohammad S. "A literature survey on algorithms for multi-label learning." Oregon State University, Corvallis 18 (2010).

  • $\begingroup$ Although this is an interesting thread, it does not really help in my situation, because that thread is about single-label classification. Like they state: "we assign equal cost to false positives and false negatives". This is exactly not, what we have here, because the definition only assigns cost to false negatives, not to false positives.. $\endgroup$
    – ma0ho
    Jul 18, 2018 at 14:22
  • $\begingroup$ My answer at the duplicate thread applies equally well to multi-label classification. $\endgroup$ Jul 18, 2018 at 15:23


Browse other questions tagged or ask your own question.