8
$\begingroup$

I'm classifying a data set using SVM and those are the precision and recall values for two classes.

     precision    recall  f1-score   support

H       0.91      0.99      0.95      1504
R       0.81      0.23      0.36       192

avg/total 0.90      0.91      0.88      1696

Following is the confusion matrix.

Confusion Matrix:
[[1494   148]
 [ 10    44]]

enter image description here Can I say this is a good classifier for my dataset based on average? I'm not sure because I'm getting a low recall value for class 'R'?

$\endgroup$
0

2 Answers 2

12
$\begingroup$

The quality of your classifier, as those metrics show, will depend on how you intend to use it. E.g.

  • It is a great classifier if you data is a set of documents, if you are looking for documents of type H, and you're main concern is to make sure that most relevant documents are retrieved (high recall on H). Furthermore, the precision of H, i.e. the percentage of retrieved documents are relevant, is high too, so that's even better in case having irrelevant document amongst the retrieved documents is costly.
  • It is a terrible classifier if you try to retrieve as many documents of type R as possible, because the recall on R is 0.23 only, which means you are going to miss 77% of the documents.
  • It is a great classifier if you want to retrieve just a few documents of type R (low recall on R doesn't matter in this case) but having irrelevant document amongst the retrieved documents is costly (since you have a high precision on R, you won't have to pay too much for irrelevant documents).
  • etc.

(Btw there is unfortunately no consensus on the confusion matrix notation, so when you post a conversion matrix, you might want to specify where the predicted and true values are, even though in most cases, we can infer it from the precision/recall values)

$\endgroup$
2
  • $\begingroup$ Hi Franck, Thanks for your answer and for the advice on Confusion Matrix :). In my case its highly important to identify both 'R' and 'H' accurately. According to you this is not good in that, since i'm getting low recall for class 'R',right? $\endgroup$ Jul 23, 2014 at 4:55
  • $\begingroup$ @NilaniAlgiriyage Yes, issue to identify R indeed! $\endgroup$ Jul 23, 2014 at 5:01
4
$\begingroup$

One difficulty in answering the question is that you didn't mention what is the nature of the data set that you are actually using the classifier for. Franck's answer is excellent, but he assumes you are using it to find documents. The response may be a bit different if your application is medical research / clinical trials or evaluating the performance of your SVM for a stock or futures trading system, etc.

In your case the support values for H & R are very different and this has implications for the bias inherent in metrics such as Precision, Recall and F1 that you are using.

Depending on the application, it may be preferable to use unbiased metrics which adjust for the differences in support. For example, the unbiased version of Precision is Markedness, defined as: Markedness = Precision + NPV - 1 = TP/(TP+FP) +FN/(FN+TP) = 0.72 for your example. The other unbiased metric is Informedness = TPR - FPR = distance from random on the ROC chart, which comes out to be 0.22 in your case. The geometric mean of Markedness & Informedness is the Matthews correlation coefficient = 0.40 for your example.

Whether or not the low value of Sensitivity (Recall) for class R is a problem depends on the associated "cost" of this error in your particular case.

$\endgroup$
1
  • $\begingroup$ Thanks Tony for your explanation. FYI-my application area is web usage mining. $\endgroup$ Jul 23, 2014 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.