20
$\begingroup$

Precision is defined as:

p = true positives / (true positives + false positives)

Is it correct that, as true positives and false positives approach 0, the precision approaches 1?

Same question for recall:

r = true positives / (true positives + false negatives)

I am currently implementing a statistical test where I need to calculate these values, and sometimes it happens that the denominator is 0, and I am wondering which value to return for this case.

P.S.: Excuse the inappropriate tag, I wanted to use recall, precision and limit, but I cannot create new Tags yet.

$\endgroup$
  • $\begingroup$ I don't think we need limit tag. $\endgroup$ – user88 Aug 17 '10 at 11:08
  • $\begingroup$ Presumably you're attempting to quantify performance of some diagnostic procedure; is there any reason you're not using a proper signal detection theory metric like d', A', or area under the ROC curve? $\endgroup$ – Mike Lawrence Aug 17 '10 at 11:19
  • 3
    $\begingroup$ @Mike, precision and recall are common evaluation metrics in, e.g., information retrieval where ROC, or in particular specificity is awkward to use because you already expect a high number of false positives. $\endgroup$ – user979 Aug 17 '10 at 18:02
17
$\begingroup$

Given a confusion matrix:

            predicted
            (+)   (-)
            ---------
       (+) | TP | FN |
actual      ---------
       (-) | FP | TN |
            ---------

we know that:

Precision = TP / (TP + FP)
Recall = TP / (TP + FN)

Lets consider the cases where the denominator is zero:

  • TP+FN=0 : means that there were no positive cases in the input data
  • TP+FP=0 : means that all instances were predicted as negative
$\endgroup$
  • 9
    $\begingroup$ Extending your answer: If TP=0 (as in both cases), recall is 1, since the method has discovered all of none true positives; precision is 0 if there is any FP and 1 otherwise. $\endgroup$ – user88 Aug 17 '10 at 11:44
11
$\begingroup$

Answer is Yes. The undefined edge cases occur when true positives (TP) are 0 since this is in the denominator of both P & R. In this case,

  • Recall = 1 when FN=0, since 100% of the TP were discovered
  • Precision = 1 when FP=0, since no there were no spurious results

This is a reformulation of @mbq's comment.

$\endgroup$
3
$\begingroup$

I am familiar with different terminology. What you call precision I would positive predictive value (PPV). And what you call recall I would call sensitivity (Sens). :

http://en.wikipedia.org/wiki/Receiver_operating_characteristic

In the case of sensitivity (recall), if the denominator is zero (as Amro points out), there are NO positive cases, so the classification is meaningless. (That does not stop either TP or FN being zero, which would result in a limiting sensitivity of 1 or 0. These points are respectively at the top right and bottom left hand corners of the ROC curve - TPR = 1 and TPR = 0.)

The limit of PPV is meaningful though. It is possible for the test cut-off to be set so high (or low) so that all cases are predicted as negative. This is at the origin of the ROC curve. The limiting value of the PPV just before the cutoff reaches the origin can be estimated by considering the final segment of the ROC curve just before the origin. (This may be better to model as ROC curves are notoriously noisy.)

For example if there are 100 actual positives and 100 actual negatives and the final segnemt of the ROC curve approaches from TPR = 0.08, FPR = 0.02, then the limiting PPV would be PPR ~ 0.08*100/(0.08*100 + 0.02*100) = 8/10 = 0.8 i.e 80% probability of being a true positive.

In practice each sample is represented by a segment on the ROC curve - horizontal for an actual negative and vertical for an actual positive. One could estimate the limiting PPV by the very last segment before the origin, but that would give an estimated limiting PPV of 1, 0 or 0.5, depending on whether the last sample was a true positive, a false positive (actual negative) or made of an equal TP and FP. A modelling approach would be better, perhaps assuming the data are binormal - a common assumption, eg: http://mdm.sagepub.com/content/8/3/197.short

$\endgroup$
1
$\begingroup$

That would depend on what you mean by "approach 0". If false positives and false negatives both approach zero at a faster rate than true positives, then yes to both questions. But otherwise, not necessarily.

$\endgroup$
  • $\begingroup$ I really do not know the rate. To be honest all I know is that my program crashed with a division-by-zero and that I need to handle that case somehow. $\endgroup$ – Björn Pollex Aug 17 '10 at 9:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.