Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 18 down vote favorite
9

I'm wondering how to calculate precision and recall measures for multiclass multilabel classification, i.e. classification where there are more than two labels, and where each instance can have multiple labels?

share|improve this question

3 Answers

up vote 5 down vote

Another popular tool for measuring classifier performance is ROC/AUC ; this one too has a multi-class / multi-label extension : see [Hand 2001]

[Hand 2001]: A simple generalization of the area under the ROC curve to multiple class classification problems

share|improve this answer
up vote 1 down vote

I don't know about the multi-label part but for the mutli-class classification those links will help you

This link explains how to build the confusion matrix that you can use to calculate the precision and recall for each category

And this link explains how to calculate micro-f1 and macro-f1 measures to evaluate the classifier as a whole.

hope that you found that useful.

share|improve this answer
1  
The key point is: there are multiple possible valid ways to compute these metrics (e.g., micro-F1 vs macro-F1) because there are multiple ways to define what is correct. This depends on your application and validity criteria. – Jack Tanner Jan 28 '12 at 1:57
Ahmed: Thanks for the links! @JackTanner Would you perhaps have a reference for this (for the case of multi-class multi-label classification)? – MaVe Jan 28 '12 at 7:59
@MaVe, sorry, no links. This is just from personal experience. You'll get there simply by thinking about what constitutes, say, a true positive and a false positive for your purposes. – Jack Tanner Jan 28 '12 at 14:11
up vote 0 down vote

Check out these slides from cs205.org at Harvard. Once you get to the section on Error Measures, there is discussion of precision and recall in multi-class settings (e.g., one-vs-all or one-vs-one) and confusion matrices. Confusion matrices is what you really want here.

FYI, in the Python software package scikits.learn, there are built-in methods to automatically compute things like the confusion matrix from classifiers trained on multi-class data. It can probably directly compute precision-recall plots for you too. Worth a look.

share|improve this answer
1  
Unfortunately the link to the slides is dead and I could find the slides elsewhere. – f3lix Oct 5 '12 at 11:55
It will replenish when they get to that lecture in this year's course. If I could copy the PDF to a permanent link location, I would, but I can't, so it periodically breaking is unavoidable and there won't be any other place to find the notes, they are specific to that course. – EMS Oct 5 '12 at 12:37

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.