2
$\begingroup$

I am really confused about how to calculate precision and recall in supervised machine learning algorithm using NB classifier with more than two classes.

Say for example

  1. I have three classes $A$, $B$, & $C$
  2. I have $10000$ Documents out of which $2000$ goes to training sample set (class $A=500$, class $B=1000$, class $C=500$)
  3. Now on basis of above training sample set classify rest $8000$ documents using NB classifier
  4. After classifying, $1000$ documents goes to class $A$ and $6000$ documents goes to class $B$ and $1000$ documents goes to $C$
  5. Now how to calculate precision and recall for all individual classes?

I figured out precision and recall for two classes here it goes

Say suppose there are two classes $A$, $B$

Now when a test is executed for documents labeled as $A$ there are two possible classifications for each document: if the classification is $A$, add 1 to “true A” (TA), if the classification is $B$ add 1 to “false B” (FB). Similarly for $B$: if the classification is $A$, add 1 to “false A” (FA) and if classification is B add 1 to “true B” (TB).

I want the same above situation when there are more than two classes

$\endgroup$
  • 1
    $\begingroup$ Are these ordered classes or not? Is your issue with developing a prediction from probabilistic classifications? Or do you want to maintain the probabilistic structure of the NB model and calculate a weighted sensitivity/specificity for your model? $\endgroup$ – AdamO Jul 5 '13 at 18:37
  • $\begingroup$ By "NB classifier", do you mean naive Bayes? $\endgroup$ – gung Jul 5 '13 at 19:00
  • $\begingroup$ I think the question is much more basic than that. $\endgroup$ – Gala Jul 5 '13 at 19:02
  • $\begingroup$ See also stats.stackexchange.com/questions/51296/… $\endgroup$ – Gala Jul 8 '13 at 6:42
7
$\begingroup$

The logic remains the same for several classes, to wit

  • If a document belonging to A…

    • is classified as A, it's a true positive/true A
    • is classified as B, it's a false positive for B/false B and a false negative for A
    • is classified as C, it's a false positive for C/false C and a false negative for A
  • If a document belonging to B…

    • is classified as A, it's a false positive for A/false A and a false negative for B
    • is classified as B, it's a true positive for B
    • is classified as C, it's a false positive for C/false C and a false negative for B
  • etc.

Precision for A is true positives/(true positives + false positives) where “false positives” are the false positives from all other classes (i.e. the B documents classified as A + the C documents classified as A, etc.).

Recall for A is true positives/(true positives + false negatives) where “false negatives” are all the A documents not classified as A (i.e. the A documents classified as B + the A documents classified as C, etc.) or, equivalently, the total number of A documents minus the number of true positives.

You can also look at all this as a series of confusion matrices with two categories: One with A and non-A (so B and C together), one with B and non-B and finally one with C and non-C.

Most informative is to report precision and recall for each category (especially if you have just a few) but I have seen people combine them in a F1 score and average across categories to obtain some sort of overall performance measure.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.