29
$\begingroup$

Consider a scenario where you are provided with KnownLabel Matrix and PredictedLabel matrix. I would like to measure the goodness of the PredictedLabel matrix against the KnownLabel Matrix.

But the challenge here is that KnownLabel Matrix have few rows only one 1 and other few rows have many 1's (those instance are multi labeled). An example of KnownLabel Matrix is given below.

A =[1 0 0 0
    0 1 0 0
    0 1 1 0
    0 0 1 1
    0 1 1 1]

In the above matrix, data instance 1 and 2 are single label data, data instance 3 and 4 are two label data and data instance 5 is the three label data.

Now I have PredictedLabel Matrix of data instance using an algorithm.

I would like to know various measure which can be used to measure the goodness of the PredictedLabel Matrix against KnownLabel Matrix.

I can think of frobeinus norm difference between them as one of the measure. But im looking for the measure such as accuracy $(= \frac{\text{Correctly_predicted_instance}}{\text{total_instance}})$

Here how do can we define the $\rm Correctly\_predicted$ for multiple data instance?

$\endgroup$
  • 5
    $\begingroup$ (+1) Sidenote: Is there a specific reason that you haven't accepted an answer in the majority of your questions ? Why didn't you post a comment when the provided answer did not solve your problem ? E.g.: stats.stackexchange.com/questions/9947/… $\endgroup$ – steffen Jul 6 '11 at 11:44
28
$\begingroup$

(1) gives a nice overview:

enter image description here

enter image description here

The Wikipedia page n multi-label classification contains a section on the evaluation metrics as well.

I would add a warning that in the multilabel setting, accuracy is ambiguous: it might either refer to the exact match ratio or the Hamming score (see this post). Unfortunately, many papers use the term "accuracy".


(1) Sorower, Mohammad S. "A literature survey on algorithms for multi-label learning." Oregon State University, Corvallis (2010).

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Do these definitions go against the general definitions for Precision and Recall? I have always read that precision should divide by TP+FP and recall should divide by TP+FN (the proposed definitions here do the opposite if I understood well). $\endgroup$ – tomasyany Jul 24 '17 at 14:56
  • $\begingroup$ No, the definitions in the paper are correct. Here $Y_i \in \mathcal{Y} = \{0, 1\}^k$ is a ground truth vector of labels for $i$th sample, and $Z_i = h(\mathbf{x}_i) = \{0, 1\}^k$ is a predicted set of labels as $h$ denotes a multi-label classifier. Perhaps, you've mistakenly mixed up $Y_i$ and $Z_i$ meanings. $\endgroup$ – constt Sep 7 '17 at 2:41
  • $\begingroup$ for the accuracy measure, how do you elegantly handle cases where the denominator |Y + Z| == 0? $\endgroup$ – ihadanny Jul 3 '18 at 5:57
  • 3
    $\begingroup$ @tomasyany is referring to the text definitions (not the formulas), which do appear to be switched around. $\endgroup$ – Narfanar Dec 28 '18 at 9:07
  • $\begingroup$ And this AP definition looks more like mAP (mean AP), no? What's referred to as 'Accuracy' is the average IoU. The terms are quite a bit confusing overall. $\endgroup$ – Narfanar Dec 28 '18 at 9:58
8
$\begingroup$

The Hamming Loss is probably the most widely used loss function in multi-label classification.

Have a look at Empirical Studies on Multi-label Classification and Multi-Label Classification: An Overview, both of which discuss this.

| cite | improve this answer | |
$\endgroup$
4
$\begingroup$

Correctly Predicted is the intersection between the set of suggested labels and the set expected one. Total Instances is the union of the sets above (no duplicate count).

So given a single example where you predict classes A, G, E and the test case has E, A, H, P as the correct ones you end up with Accuracy = Intersection{(A,G,E), (E,A,H,P)} / Union{(A,G,E), (E,A,H,P)} = 2 / 5

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.