I wonder how to compute precision and recall using a confusion matrix for a multi-class classification problem. Specifically, an observation can only be assigned to its most probable class / label. I would like to compute:

  • Precision = TP / (TP+FP)
  • Recall = TP / (TP+FN)

for each class, and then compute the micro-averaged F-measure.

up vote 58 down vote accepted

In a 2-hypothesis case, the confusion matrix is usually:

       | Declare H1  |  Declare H0 |
|Is H1 |    TP       |   FN        |
|Is H0 |    FP       |   TN        |

where I've used something similar to your notation:

  • TP = true positive (declare H1 when, in truth, H1),
  • FN = false negative (declare H0 when, in truth, H1),
  • FP = false positive
  • TN = true negative

From the raw data, the values in the table would typically be the counts for each occurrence over the test data. From this, you should be able to compute the quantities you need.

Edit

The generalization to multi-class problems is to sum over rows / columns of the confusion matrix. Given that the matrix is oriented as above, i.e., that a given row of the matrix corresponds to specific value for the "truth", we have:

$\text{Precision}_{~i} = \cfrac{M_{ii}}{\sum_j M_{ji}}$

$\text{Recall}_{~i} = \cfrac{M_{ii}}{\sum_j M_{ij}}$

That is, precision is the fraction of events where we correctly declared $i$ out of all instances where the algorithm declared $i$. Conversely, recall is the fraction of events where we correctly declared $i$ out of all of the cases where the true of state of the world is $i$.

  • 1
    In my case, there are 10+ classes, so I guess the FN will mean the total count of declare class H(i), i != 1; and the same is FP? – daiyue Mar 4 '13 at 17:14
  • Hi, I wonder what the values will be for the Precision and Recall, if TP+FP=0, and TP+FN=0 for some actual class in the confusion matrix. – daiyue Mar 13 '13 at 15:35
  • The precision for class i is undefined if there are no instances where the algorithm declares i. The recall for class i is undefined if the test set does not include class i. – Dave Mar 13 '13 at 18:27
  • My final goal is calculate the Macro F Measure, so I need precision and recall values for each class i; so how can I compute the Macro-F measure if the above two cases appear in some class i? In particular, what's the value for Fi, and does class i count as one of the classes M, that the number of elements in M will be count as the denominator of the formula for calculating Macro F measure. – daiyue Mar 13 '13 at 20:18
  • 1
    sry, could you explain your idea more clearly? – daiyue Mar 13 '13 at 22:54

Good summary paper, looking at these metrics for multi-class problems:

  • Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing and Management, 45, p. 427-437. (pdf)

The abstract reads:

This paper presents a systematic analysis of twenty four performance measures used in the complete spectrum of Machine Learning classification tasks, i.e., binary, multi-class, multi-labelled, and hierarchical. For each classification task, the study relates a set of changes in a confusion matrix to specific characteristics of data. Then the analysis concentrates on the type of changes to a confusion matrix that do not change a measure, therefore, preserve a classifier’s evaluation (measure invariance). The result is the measure invariance taxonomy with respect to all relevant label distribution changes in a classification problem. This formal analysis is supported by examples of applications where invariance properties of measures lead to a more reliable evaluation of classifiers. Text classification supplements the discussion with several case studies.

  • 2
    Welcome to the site, @JamesTaylor. Would you mind giving a summary of the info in the linked paper to help readers decide whether it's what they need & in case the link goes dead? – gung Mar 12 '13 at 16:27
  • 2
    That paper is a great reference. Thank you! – Saggy Manatee And Swan Folk Jan 31 '14 at 19:39

This docx, Evaluating a classification model – What does precision and recall tell me?, from Compumine provides a simple introduction to the confusion matrix and the measures derived from it.

It helps to create the confusion matrix, precision, recall, specificity and accuracy.

  • 3
    Welcome to the site, @Jayran Choopan. Would you mind giving a summary of the info in the linked paper to help readers decide whether it's what they need & in case the link goes dead? – gung Jul 8 '13 at 4:01
  • 4
    PDF link is dead – Trenton Jun 11 '15 at 3:52

Using sklearn and numpy:

from sklearn.metrics import confusion_matrix
import numpy as np

labels = ...
predictions = ...

cm = confusion_matrix(labels, predictions)
recall = np.diag(cm) / np.sum(cm, axis = 1)
precision = np.diag(cm) / np.sum(cm, axis = 0)

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.