Skip to main content
Cross Validated

Return to Answer

improve formatting
Source Link
edit approved Jun 3 at 18:41
Michael Mior
edit approved Jun 3 at 18:41
Michael Mior
  • 111
  • 4

When evaluating a classifier at high thresholds, the precision might (often actually) not be 1 when recall is 0. It's usually N/A! I think there is something wrong about how people plot the P/R curve. Avoiding N/A samples is a bias in the sense that you avoid singularity samples. 

I computed the average precision wrt to the average recall ignoring N/A samples and I never got a classifier starting at 1 for 0 recall for a shallow neural net in object detection. This was also true for curves computed with the tp,fp,fn numbers. It's quite easy to verify by paper and pencil with a single image. For example: I have a classifier that outputs for a single image: preds=[.7 .6 .5 .1 .05] truth=[n y n n y ] By

preds=[.7 .6 .5 .1 .05]
truth=[n  y   n  n  y ]

By computing the confusion matrices with the various thresholds we have: tp=[2 1 1 1 0 0],fn=[0 1 1 1 2 2],fp=[3 3 2 1 1 0]. the

tp=[2 1 1 1 0 0]
fn=[0 1 1 1 2 2]
fp=[3 3 2 1 1 0]

then recall rec=[1 .5 .5 .5 0 0]rec=[1 .5 .5 .5 0 0], and the precision=[.4 .25 1/3 precision .5 0 NaN]prec=[.4 .25 1/3 .5 0 NaN]. I don't see how it would make sense to replace a NaN or the precision(@recall==0) with 1. 1 should be an upper bound, not a value we replace precision(@recall==0) with.

When evaluating a classifier at high thresholds, the precision might (often actually) not be 1 when recall is 0. It's usually N/A! I think there is something wrong about how people plot the P/R curve. Avoiding N/A samples is a bias in the sense that you avoid singularity samples. I computed the average precision wrt to the average recall ignoring N/A samples and I never got a classifier starting at 1 for 0 recall for a shallow neural net in object detection. This was also true for curves computed with the tp,fp,fn numbers. It's quite easy to verify by paper and pencil with a single image. For example: I have a classifier that outputs for a single image: preds=[.7 .6 .5 .1 .05] truth=[n y n n y ] By computing the confusion matrices with the various thresholds we have: tp=[2 1 1 1 0 0],fn=[0 1 1 1 2 2],fp=[3 3 2 1 1 0]. the recall rec=[1 .5 .5 .5 0 0], and the precision=[.4 .25 1/3 .5 0 NaN]. I don't see how it would make sense to replace a NaN or the precision(@recall==0) with 1. 1 should be an upper bound, not a value we replace precision(@recall==0) with.

When evaluating a classifier at high thresholds, the precision might (often actually) not be 1 when recall is 0. It's usually N/A! I think there is something wrong about how people plot the P/R curve. Avoiding N/A samples is a bias in the sense that you avoid singularity samples. 

I computed the average precision wrt to the average recall ignoring N/A samples and I never got a classifier starting at 1 for 0 recall for a shallow neural net in object detection. This was also true for curves computed with the tp,fp,fn numbers. It's quite easy to verify by paper and pencil with a single image. For example: I have a classifier that outputs for a single image:

preds=[.7 .6 .5 .1 .05]
truth=[n  y   n  n  y ]

By computing the confusion matrices with the various thresholds we have:

tp=[2 1 1 1 0 0]
fn=[0 1 1 1 2 2]
fp=[3 3 2 1 1 0]

then recall rec=[1 .5 .5 .5 0 0], and precision prec=[.4 .25 1/3 .5 0 NaN]. I don't see how it would make sense to replace a NaN or the precision(@recall==0) with 1. 1 should be an upper bound, not a value we replace precision(@recall==0) with.

Post Undeleted by Jonathan Blanchette
Post Deleted by Jonathan Blanchette
Source Link
Jonathan Blanchette
Jonathan Blanchette
  • 51
  • 1
  • 1

When evaluating a classifier at high thresholds, the precision might (often actually) not be 1 when recall is 0. It's usually N/A! I think there is something wrong about how people plot the P/R curve. Avoiding N/A samples is a bias in the sense that you avoid singularity samples. I computed the average precision wrt to the average recall ignoring N/A samples and I never got a classifier starting at 1 for 0 recall for a shallow neural net in object detection. This was also true for curves computed with the tp,fp,fn numbers. It's quite easy to verify by paper and pencil with a single image. For example: I have a classifier that outputs for a single image: preds=[.7 .6 .5 .1 .05] truth=[n y n n y ] By computing the confusion matrices with the various thresholds we have: tp=[2 1 1 1 0 0],fn=[0 1 1 1 2 2],fp=[3 3 2 1 1 0]. the recall rec=[1 .5 .5 .5 0 0], and the precision=[.4 .25 1/3 .5 0 NaN]. I don't see how it would make sense to replace a NaN or the precision(@recall==0) with 1. 1 should be an upper bound, not a value we replace precision(@recall==0) with.