I'm wondering how to calculate precision and recall measures for multiclass multilabel classification, i.e. classification where there are more than two labels, and where each instance can have multiple labels?

  • 1
    $\begingroup$ the multilabel part makes it much harder and I too am interested in this. I think that it's not applicable to multilabel problems but don't trust me at all. $\endgroup$
    – user798719
    May 25, 2017 at 3:43
  • 1
    $\begingroup$ In fact, all multilabel problems are multiclass, so you can use the utiml package in R for instance or Mulan in Java. $\endgroup$ Oct 20, 2017 at 8:14

9 Answers 9


For multi-label classification you have two ways to go First consider the following.

  • $n$ is the number of examples.
  • $Y_i$ is the ground truth label assignment of the $i^{th}$ example..
  • $x_i$ is the $i^{th}$ example.
  • $h(x_i)$ is the predicted labels for the $i^{th}$ example.

Example based

The metrics are computed in a per datapoint manner. For each predicted label its only its score is computed, and then these scores are aggregated over all the datapoints.

  • Precision = $\frac{1}{n}\sum_{i=1}^{n}\frac{|Y_{i}\cap h(x_{i})|}{|h(x_{i})|}$ , The ratio of how much of the predicted is correct. The numerator finds how many labels in the predicted vector has common with the ground truth, and the ratio computes, how many of the predicted true labels are actually in the ground truth.
  • Recall = $\frac{1}{n}\sum_{i=1}^{n}\frac{|Y_{i}\cap h(x_{i})|}{|Y_{i}|}$ , The ratio of how many of the actual labels were predicted. The numerator finds how many labels in the predicted vector has common with the ground truth (as above), then finds the ratio to the number of actual labels, therefore getting what fraction of the actual labels were predicted.

There are other metrics as well.

Label based

Here the things are done labels-wise. For each label the metrics (eg. precision, recall) are computed and then these label-wise metrics are aggregated. Hence, in this case you end up computing the precision/recall for each label over the entire dataset, as you do for a binary classification (as each label has a binary assignment), then aggregate it.

The easy way is to present the general form.

This is just an extension of the standard multi-class equivalent.

  • Macro averaged $\frac{1}{q}\sum_{j=1}^{q}B(TP_{j},FP_{j},TN_{j},FN_{j})$

  • Micro averaged $B(\sum_{j=1}^{q}TP_{j},\sum_{j=1}^{q}FP_{j},\sum_{j=1}^{q}TN_{j},\sum_{j=1}^{q}FN_{j})$

Here the $TP_{j},FP_{j},TN_{j},FN_{j}$ are the true positive, false positive, true negative and false negative counts respectively for only the $j^{th}$ label.

Here $B$ stands for any of the confusion-matrix based metric. In your case you would plug in the standard precision and recall formulas. For macro average you pass in the per label count and then sum, for micro average you average the counts first, then apply your metric function.

You might be interested to have a look into the code for the mult-label metrics here , which a part of the package mldr in R. Also you might be interested to look into the Java multi-label library MULAN.

This is a nice paper to get into the different metrics: A Review on Multi-Label Learning Algorithms

  • $\begingroup$ It would have been good if you could have given references regarding the authenticity of the metrics that you have written like some wikipedia reference. The references that you have mentioned are the coding part of the metrics or research paper.. $\endgroup$
    – Rajat Jain
    Mar 6, 2019 at 9:53
  • 6
    $\begingroup$ A review paper is already linked at the end of the answer (A Review on Multi-Label Learning Algorithms) ieeexplore.ieee.org/document/6471714 . These are well known metrics in the literature based on which the implementations are done. I am not sure how I can demonstrate authenticity. $\endgroup$
    – phoxis
    Mar 6, 2019 at 20:31
  • 1
    $\begingroup$ +1. I'm still confused about how you would count, e.g., TP for this. (sorry but that reference is paywalled). So if for one label in one example, you had a ground truth = 2 and prediction = 3, do you have 2 TP and 1 FP for that example and for that label? $\endgroup$
    – mikey
    Dec 17, 2021 at 11:02

Another popular tool for measuring classifier performance is ROC/AUC ; this one too has a multi-class / multi-label extension : see [Hand 2001]

[Hand 2001]: A simple generalization of the area under the ROC curve to multiple class classification problems

  • $\begingroup$ It is popular, but it can be buggy. I don't entirely trust it. stats.stackexchange.com/questions/93901/… $\endgroup$ Oct 29, 2014 at 16:14
  • 6
    $\begingroup$ Never change stack overflow! Guy asks a problem, most voted answer doesn't actually answer his question, but points out some other tool/library which would be better $\endgroup$
    – humble
    May 23, 2018 at 9:38
  • $\begingroup$ Yes, how can this answer have +20? It doesn't even contain the words precision and recall. $\endgroup$
    – Simon
    Jan 21, 2019 at 13:54
  • 2
    $\begingroup$ if you think thoroughly you will realize that precision and recall are actually captured by AUC. $\endgroup$
    – oDDsKooL
    Jan 30, 2019 at 5:21
  • 2
    $\begingroup$ If you think thoroughly you will realise that you cannot recover both precision and recall from a single value like AUC; it's often important to know both, and the question asks for both. $\endgroup$
    – Denziloe
    Sep 18, 2021 at 16:54

Here is some discuss of coursera forum thread about confusion matrix and multi-class precision/recall measurement.

The basic idea is to compute all precision and recall of all the classes, then average them to get a single real number measurement.

Confusion matrix make it easy to compute precision and recall of a class.

Below is some basic explain about confusion matrix, copied from that thread:

A confusion matrix is a way of classifying true positives, true negatives, false positives, and false negatives, when there are more than 2 classes. It's used for computing the precision and recall and hence f1-score for multi class problems.

The actual values are represented by columns. The predicted values are represented by rows.


10 training examples that are actually 8, are classified (predicted) incorrectly as 5
13 training examples that are actually 4, are classified incorrectly as 9

Confusion Matrix

cm =
     0     1     2       3     4       5       6     7      8       9      10
     1   298     2       1     0       1       1     3      1       1       0
     2     0     293     7     4       1       0     5      2       0       0
     3     1     3      263    0       8       0     0      3       0       2
     4     1     5       0     261     4       0     3      2       0       1
     5     0     0       10    0     254       3     0     10       2       1
     6     0     4       1     1       4       300   0      1       0       0
     7     1     3       2     0       0       0     264    0       7       1
     8     3     5       3     1       7       1     0      289     1       0
     9     0     1       3     13      1       0     11     1       289     0
    10     0     6       0     1       6       1     2      1       4       304

For class x:

  • True positive: diagonal position, cm(x, x).

  • False positive: sum of column x (without main diagonal), sum(cm(:, x))-cm(x, x).

  • False negative: sum of row x (without main diagonal), sum(cm(x, :), 2)-cm(x, x).

You can compute precision, recall and F1 score following course formula.

Averaging over all classes (with or without weighting) gives values for the entire model.

  • 4
    $\begingroup$ You have your axes flipped. Based on what you have written, your CM should be transposed. $\endgroup$
    – Spacey
    Jan 29, 2017 at 19:28
  • $\begingroup$ @Tarantula Why do you think so? I think he is correct. $\endgroup$
    – shahensha
    Feb 17, 2017 at 2:17
  • $\begingroup$ @shahensha Try it out for one column, it's wrong. $\endgroup$
    – Spacey
    Feb 18, 2017 at 23:32
  • 1
    $\begingroup$ The link to the Coursera thread is broken $\endgroup$
    – shark8me
    Apr 26, 2017 at 5:12
  • 11
    $\begingroup$ i do not believe that this answer handles the multilabel problem. it applies to multi class problems. What is the notion of a false positive or false negative in multilabel problems? $\endgroup$
    – user798719
    May 25, 2017 at 3:41

I don't know about the multi-label part but for the mutli-class classification those links will help you

This link explains how to build the confusion matrix that you can use to calculate the precision and recall for each category

And this link explains how to calculate micro-f1 and macro-f1 measures to evaluate the classifier as a whole.

hope that you found that useful.

  • 5
    $\begingroup$ The key point is: there are multiple possible valid ways to compute these metrics (e.g., micro-F1 vs macro-F1) because there are multiple ways to define what is correct. This depends on your application and validity criteria. $\endgroup$ Jan 28, 2012 at 1:57
  • $\begingroup$ Ahmed: Thanks for the links! @JackTanner Would you perhaps have a reference for this (for the case of multi-class multi-label classification)? $\endgroup$
    – Vam
    Jan 28, 2012 at 7:59
  • 1
    $\begingroup$ @MaVe, sorry, no links. This is just from personal experience. You'll get there simply by thinking about what constitutes, say, a true positive and a false positive for your purposes. $\endgroup$ Jan 28, 2012 at 14:11
  • 4
    $\begingroup$ First link died $\endgroup$ Aug 3, 2013 at 1:22

Exactly the same way you would do it general case, with sets:



Here are simple Python functions that do exactly that:

def precision(y_true, y_pred):
    i = set(y_true).intersection(y_pred)
    len1 = len(y_pred)
    if len1 == 0:
        return 0
        return len(i) / len1

def recall(y_true, y_pred):
    i = set(y_true).intersection(y_pred)
    return len(i) / len(y_true)

def f1(y_true, y_pred):
    p = precision(y_true, y_pred)
    r = recall(y_true, y_pred)
    if p + r == 0:
        return 0
        return 2 * (p * r) / (p + r)

if __name__ == '__main__':
    print(f1(['A', 'B', 'C'], ['A', 'B']))

this link helped me.. https://www.youtube.com/watch?v=HBi-P5j0Kec i am hoping it will help you as well

say the distribution as as below

    A   B   C   D
A   100 80  10  10
B   0    9   0   1
C   0    1   8   1
D   0    1   0   9

the precision for A would be

P(A) = 100/ 100 + 0 + 0 +0 = 100

P(B) = 9/ 9 + 80 + 1 + 1 = 9/91 psst... essentially take the true positive of the class and divide up by the column data across rows

recall for a would be

R(A) = 100/ 100+ 80+10+10 = 0.5

R(B) = 9 / 9+ 0+0+1 = 0.9

psst... essentially take the true positive of the class and divide up by the row data across columns

once you get all the values, take the macro average

avg(P) = P(A) + P(B) + P(C) + P(D) / 4

avg(R) = R(A) + R(B) + R(C) + R(D) / 4

F1 = 2 *avg(P) * avg(R) / avg(P) + avg(R)


Check out these slides from cs205.org at Harvard. Once you get to the section on Error Measures, there is discussion of precision and recall in multi-class settings (e.g., one-vs-all or one-vs-one) and confusion matrices. Confusion matrices is what you really want here.

FYI, in the Python software package scikits.learn, there are built-in methods to automatically compute things like the confusion matrix from classifiers trained on multi-class data. It can probably directly compute precision-recall plots for you too. Worth a look.

  • 4
    $\begingroup$ Unfortunately the link to the slides is dead and I could find the slides elsewhere. $\endgroup$
    – f3lix
    Oct 5, 2012 at 11:55
  • $\begingroup$ It will replenish when they get to that lecture in this year's course. If I could copy the PDF to a permanent link location, I would, but I can't, so it periodically breaking is unavoidable and there won't be any other place to find the notes, they are specific to that course. $\endgroup$
    – ely
    Oct 5, 2012 at 12:37
  • $\begingroup$ sklearn doesn't support multi-label for confusion matrix github.com/scikit-learn/scikit-learn/issues/3452 $\endgroup$ Dec 11, 2015 at 2:52

From Ozgur et al (2005) it is possible to see that you should compute Precision and Recall following the normal expressions, but instead of averaging over total N instances in your dataset, you should use N=[instances with at least one label with the class in question assigned to].

here is the reference mentioned: http://citeseerx.ist.psu.edu/viewdoc/download?doi=


In case if you want to see the results directly:

from sklearn.metrics import classification_report, confusion_matrix

classification_report(y_test, y_pred)

This would work in case you want average precision, recall and f-1 score

 from sklearn.metrics import precision_recall_fscore_support as score
 print ('Precision : {}'.format(precision))
 print ('Recall    : {}'.format(recall))
 print ('F-score   : {}'.format(fscore))

Not the answer you're looking for? Browse other questions tagged or ask your own question.