0
$\begingroup$

I have an object detection problem with only one object class. I want to compare the results and thought about using precision and recall. They are defined as follows: $$precision = \frac{TP}{TP + FP} \quad$$ and $$recall = \frac{TP}{TP + FN} \quad$$ My problem is that I don't understand what true and false negatives are in my scenario. I have only one object class which needs to be detected in an image. The ground truth only consists of the object bounding box (so the true positives). The detected objects are either a true positive (Intersection over Union > 0.5) or a false positive (IoU < 0.5) am I right? So what are the negatives in my scenario or how can I calculate precision and recall without them?

$\endgroup$
4
  • $\begingroup$ Don't you have a confusion matrix of identifications of your object and non-identifications? Sometimes you have the object and identify it as the object. Sometimes you have the object but fail to identify it. Sometimes you don't have the object but identify it. Sometimes you don't have the object and don't identify it. $\endgroup$
    – Dave
    Nov 14, 2019 at 15:31
  • $\begingroup$ @Dave ah, so having a ground truth but no detection for that region would be a false negative? $\endgroup$
    – Sandrogo
    Nov 14, 2019 at 15:35
  • 2
    $\begingroup$ It sounds a bit as if all your images contained the target object, and what you are trying to do is not to decide whether the object is present or not, but where it is. Is that correct? If so, precision/recall do not make sense as KPIs. $\endgroup$ Nov 14, 2019 at 15:36
  • $\begingroup$ @S.Kolassa-ReinstateMonica that is correct. I only have bounding boxes and need to evaluate them. What do you suggest? $\endgroup$
    – Sandrogo
    Nov 14, 2019 at 15:37

2 Answers 2

1
$\begingroup$

As per the comments, your task is not object detection as such, since the target object is present in all images. Thus, precision and recall are not appropriate error measures, since they are limited to binary classification. Instead, you have bounding boxes and need to evaluate their quality.

First off, I am sure other people have thought about this. Searching for "bounding box quality measures" or similar terms should yield papers that could be inspirational (for instance, my search turned up this).

Assuming your boxes are rectangles with horizontal and vertical slides, they are defined by the bottom left and the top right corner. So one thing you could do is to assess the Euclidean distance of each of these corners to the "true" corner, then add these two distances up. If your rectangles are rotated, a similar approach could be done, but you would need to match corresponding corners (three of them, because the rotation angle introduces a new degree of freedom).

$\endgroup$
5
  • $\begingroup$ I have to edit my comment. Precision and recall are not actually limited to binary classification as you stated. To put it right: precision and recall are indeed used as quality measures in object detection as shown here. Maybe you can edit your answer. $\endgroup$
    – Sandrogo
    Nov 14, 2019 at 16:52
  • $\begingroup$ @Sandrogo: I only had time to skim that link, but it seems like precision there is indeed used for binary classification - the formula defining is is in terms of TP etc., it's the standard formula. Of course, these precisions can then be post-processed, averaged etc. Where do you see it's used for something else? $\endgroup$ Nov 15, 2019 at 16:38
  • $\begingroup$ The definitions of TP, FN etc. are used to adapt to the problem of measuring the quality of the detected bounding boxes. They are based on the IOU of the ground truth bounding box and the detected bounding box. So it is not a classification problem per se. $\endgroup$
    – Sandrogo
    Nov 16, 2019 at 12:52
  • $\begingroup$ Ah, I see. It amounts to converting the detection task into a binary task (is the Intersection Over Union, IOU, above a threshold or not). You are right that this applies the concept of accuracy to a fundamentally non-binary task. To be honest, I would rather use IOU as an evaluation metric directly to avoid all the problems with discretization (e.g., here). This approach also suffers from all the issues outlined here. $\endgroup$ Nov 16, 2019 at 14:54
  • $\begingroup$ Yes I already ran into many issues with this threshold approach. For example dealing with bounding boxes overlapping multiple ground truths. Nevertheless, I need to be able to decide if a detection was correct or not, so I came up with a different way. Roughly said I calculate a matrix of all IOU combinations between detections and ground truths and find the combinations with the highest scores. That are my true positives. Unassigned detections are false positives and so on... Problematic topic, but thanks for your help and interest! :) $\endgroup$
    – Sandrogo
    Nov 16, 2019 at 15:01
-2
$\begingroup$

False negatives would be the ground truth bounding boxes that were not identified. Ground truth boxes that do not have an IoU > 0.5 with any of the detected boxes. The image should explain this better.

This image should give a better understanding

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.