1
$\begingroup$

In my problem, it is possible to have multiple predictions for a ground truth instance. How we define precision in such scenarios?

For further clarification consider the following example. We have 1 ground truth (positive, think of it as a bounding box around a dog where you want to find such bounding boxes) based on an image and we have 4 predictions for it (4 bounding boxes), out of which 3 meet the criteria to be considered as correct prediction (match the ground truth with some IoU bigger than a threshold), and one not.

  • case one: If we interpret precision as the proportion of the true positive predictions divided by all positive predictions made by model, then we have: TP=3 and precision=0.75

  • case two: If we interpret precision as TP/(TP+FP), where TP is defined as the number of ground truth instances that are correctly predicted by model (hence TP=1) and FP is defined as the false predictions (FP=1), then it will be Precision=0.5

Which one would be the correct choice (I assume case one)? Note I do not want to perform any clustering, etc. to combine the three predictions.

$\endgroup$
1
  • $\begingroup$ What's the reason for not combining nearby predictions? $\endgroup$ Feb 17 at 20:06

1 Answer 1

0
$\begingroup$

Clearly, there is no single correct evaluation protocol. It depends on what you aim for in your detection system. The properties of the detector that you want to quantify are typically determined by use cases.

Greedy matching is commonly employed when evaluating a detection system on a single frame (a full image). The prediction with the most overlap is the only one that is matched when there are multiple predictions that correspond to the same ground truth. Any detection that is not matched is considered a false positive.

The precision in your example is $1/4$ when you apply this rule. Since 3 detections of a single dog count as 1 correct detection and 2 false detections. And the additional prediction that did not even meet the criteria causes another false detection.

This rule results in a penalty for a noisy detection system that returns multiple bounding boxes around each object. For instance, the PASCAL challenge metric implemented this rule. In addition, various metrics are employed for evaluating object detection, including average precision (AP) and mean average precision (mAP). You can take a look at review_object_detection_metrics.

The detection system should employ non-maximum suppression (NMS) to merge nearby predictions. Also a score or confidence is typically returned by the detection system for each prediction. And only predictions with a high enough score are returned.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.