Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm classifying a data set using SVM and those are the precision and recall values for two classes.

     precision    recall  f1-score   support

H       0.91      0.99      0.95      1504
R       0.81      0.23      0.36       192

avg/total 0.90      0.91      0.88      1696

Following is the confusion matrix.

Confusion Matrix:
[[1494   148]
 [ 10    44]]

enter image description here Can I say this is a good classifier for my dataset based on average? I'm not sure because I'm getting a low recall value for class 'R'?

share|improve this question

2 Answers 2

up vote 3 down vote accepted

The quality of your classifier, as those metrics show, will depend on how you intend to use it. E.g.

  • It is a great classifier if you data is a set of documents, if you are looking for documents of type H, and you're main concern is to make sure that most relevant documents are retrieved (high recall on H). Furthermore, the precision of H, i.e. the percentage of retrieved documents are relevant, is high too, so that's even better in case having irrelevant document amongst the retrieved documents is costly.
  • It is a terrible classifier if you try to retrieve as many documents of type R as possible, because the recall on R is 0.23 only, which means you are going to miss 77% of the documents.
  • It is a great classifier if you want to retrieve just a few documents of type R (low recall on R doesn't matter in this case) but having irrelevant document amongst the retrieved documents is costly (since you have a high precision on R, you won't have to pay too much for irrelevant documents).
  • etc.

(Btw there is unfortunately no consensus on the confusion matrix notation, so when you post a conversion matrix, you might want to specify where the predicted and true values are, even though in most cases, we can infer it from the precision/recall values)

share|improve this answer
    
Hi Franck, Thanks for your answer and for the advice on Confusion Matrix :). In my case its highly important to identify both 'R' and 'H' accurately. According to you this is not good in that, since i'm getting low recall for class 'R',right? –  Nilani Algiriyage Jul 23 at 4:55
    
@NilaniAlgiriyage Yes, issue to identify R indeed! –  Franck Dernoncourt Jul 23 at 5:01
    
Thanks very much! –  Nilani Algiriyage Jul 23 at 5:05

One difficulty in answering the question is that you didn't mention what is the nature of the data set that you are actually using the classifier for. Franck's answer is excellent, but he assumes you are using it to find documents. The response may be a bit different if your application is medical research / clinical trials or evaluating the performance of your SVM for a stock or futures trading system, etc.

In your case the support values for H & R are very different and this has implications for the bias inherent in metrics such as Precision, Recall and F1 that you are using.

Depending on the application, it may be preferable to use unbiased metrics which adjust for the differences in support. For example, the unbiased version of Precision is Markedness, defined as: Markedness = Precision + NPV - 1 = TP/(TP+FP) +FN/(FN+TP) = 0.72 for your example. The other unbiased metric is Informedness = TPR - FPR = distance from random on the ROC chart, which comes out to be 0.22 in your case. The geometric mean of Markedness & Informedness is the Matthews correlation coefficient = 0.40 for your example.

Whether or not the low value of Sensitivity (Recall) for class R is a problem depends on the associated "cost" of this error in your particular case.

share|improve this answer
    
Thanks Tony for your explanation. FYI-my application area is web usage mining. –  Nilani Algiriyage Jul 23 at 7:30

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.