1
$\begingroup$

I have built a classification model to recognise a class and I have evaluated it on several datasets. The problem is that some of these datasets do not have any true instance of the class in question, and so, the model should ideally recognise zero instances.

If the model has 0 false positives on such a dataset, I may claim that the precision of this model on this dataset is 100%. Conceptually, a classification model that recognises 0 instances out of 0 with no FP is a precise model. If, however, the model has only one false positive, the precision will drop down to 0 although a model with only one FP may generally be assumed to have a good precision.

With regards to the recall, since the TP and FN will always be 0, may I claim that the recall on such datasets should always be 100%?

If not, what is the appropriate way to calculate precision/recall in such cases? I am, by the way, aware of the existence of other accuracy evaluation measures but I need to calculate these two (i.e. precision and recall).

$\endgroup$
  • $\begingroup$ Precision and recall, by definition, require positives. There is no way around it without changing their definition. $\endgroup$ – Marc Claesen Aug 13 '14 at 14:09
  • $\begingroup$ In my view, I think, even without any positives, recognising 0 instances out of 0 with FP=0 is a good precision, isn't it? Precision may be defined to equal 1 if positives=0 and fp=0 and tp/(tp+fp) otherwise (I have just changed its definiton, haven't I?!). I agree with you, their definition may need to be extended or changed to account for such a possibility. $\endgroup$ – Sultan Abraham Aug 13 '14 at 14:41
  • 1
    $\begingroup$ I disagree. It just makes no sense to talk about precision and recall without positives. The reason you want to mention these is to assess the quality of a model. These measures can't give any useful information, since you can't distinguish a good model from one that always predicts negative in your case. $\endgroup$ – Marc Claesen Aug 13 '14 at 14:57
  • $\begingroup$ That is a good point. Could you please add your comments as an answer so I can accept it? $\endgroup$ – Sultan Abraham Aug 13 '14 at 15:04
3
$\begingroup$

In my opinion it makes no sense to talk about precision and recall without positives. The reason you want to mention these is to assess the quality of a model. These measures can't give any useful information, since you can't distinguish a good model from a trivial one that always predicts negative in your case.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.