7
$\begingroup$

I have an object detection algorithm for which I would like to plot an ROC curve. For this, I need the values of the fall-out corresponding to values of recall.

The false positive rate, or fall-out, is defined as $$\text{Fall-out}=\frac{FP}{FP+TN}$$

In my data, a given image may have many objects. So, almost every image has at least one box. I am counting a predicted box as a true positive if its IOU with a truth box is above a certain threshold, and as a false positive otherwise. Any truth box with no prediction box having an IOU above a threshold counts as a false negative.

However, for the denominator, how is a "true negative" defined here? In the context of object detection, what does it mean to say that something is a true negative? Can any bounding box not detected, that does not correspond to an actual object (with an IOU above a certain threshold), be called a true negative? If so, wouldn't the number of true negatives be infinite?

From what I understand, $TN$ would be the number of "non-objects" that were not detected. How can this be quantified? Is it defined in some particular way, or undefined, or infinity? Would it even be possible to plot an ROC for this?

How can I do this?

Thank you.

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ Seems to me like false positive rate does not make sense in this scenario unless you can enumerate specific objects in your images that the algorithm should not detect. Otherwise, there are conceivably thousands of boxes which would be considered true negatives. $\endgroup$
    – Joel
    Commented May 2, 2017 at 18:34
  • 1
    $\begingroup$ @Joel This was exactly what I thought. I think the number of true negatives could be infinite. Part of my question is to confirm if that is the case. Thank you for your comment. $\endgroup$
    – GoodDeeds
    Commented May 2, 2017 at 18:36
  • 2
    $\begingroup$ Also, see this question $\endgroup$
    – Joel
    Commented May 2, 2017 at 18:44

3 Answers 3

Reset to default
3
$\begingroup$

There are two errors that your algorithm can make. The first error is detecting an object when it's not there. This is the false positive. The second error is not detecting an object when it's there. This is the false negative.

To compute the false positive rate you want to compute how often it detected an object when the object was not there. This is the quotient of the false positives and all potential false positives: i.e.

$\frac{\text{detect box when no box}}{\text{all no box}}$

To compute the false negative rate you compute how often an object is not detected divided by how often the object could have been detected.

$\frac{\text{detect no box when box}}{\text{all box}}$

The key distinction between the false positive and false negative rate is which situation / hypothesis is true. False positives exist only when there are no boxes. False negatives exist only when there are boxes.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for your answer, Could you please elaborate on "False positives exist only when there are no boxes"? As my question asked how to calculate the number of "no boxes", do you mean to say that it is not possible, as the concept of a "no box" does not exist? $\endgroup$
    – GoodDeeds
    Commented May 1, 2017 at 11:00
  • $\begingroup$ False positives / negatives are errors right? So let's say your algorithm tries to detect whether there is a box in n+m pictures. In n pictures there are no boxes. In m pictures there are boxes. False positives can only occur in pictures where there are no boxes so in n pictures: i.e. when can you make an error by detecting something that's not there? False negatives can only occur in pictures where there are boxes so in m pictures. $\endgroup$
    – user31790
    Commented May 1, 2017 at 11:06
  • $\begingroup$ Thank you, I understood that statement now. Sorry, I realized that I have not been very clear, I am not sure if it has a different name, but in my case, a given image may have many objects. So, almost every image has at least one box. I am counting a predicted box as a true positive if its IOU with a truth box is above a certain threshold, and a false positive otherwise. Any truth box with no prediction box having an IOU above a threshold counts as a false negative. But what is a true negative? $\endgroup$
    – GoodDeeds
    Commented May 1, 2017 at 11:10
  • $\begingroup$ I don't think I completely understand then. One way I would tackle the problem of counting boxes in pictures - this is the problem that you're attempting to tackle right? - is to compare the number of boxes in the picture to the number of boxes that the algorithm has found and use some criterion to minimize the deviation. But I don't understand what it is that you're trying to do here? $\endgroup$
    – user31790
    Commented May 1, 2017 at 11:25
  • $\begingroup$ I am sorry for the lack of clarity. I have a set of images, and a program that detects all (possibly >1) objects, with their bounding boxes, in each of the images. I want to plot an ROC curve for this program. I think I can calculate the values of FP, FN, and TP, as mentioned in my previous comment. My question is, how do I calculate TN? From what I understand, TN would be the number of "non-objects" that were not detected. How can this be quantified? Is it defined in some particular way, or undefined, or infinity? Please let me know if I need to be clearer in explaining my problem. Thank you. $\endgroup$
    – GoodDeeds
    Commented May 1, 2017 at 11:29
0
$\begingroup$

If there are 1000 pixels in an image out of that 100 belongs to an object then if your bounding box encloses all 100 pixels, then all other 900 pixels are true negative pixels

Improve this answer
$\endgroup$
2
  • $\begingroup$ But wouldn't the hundred pixels count as one object, so contribute 1 unit to the TP? In your formulation how can you count any number of non-objects using the 900 pixels? $\endgroup$
    – GoodDeeds
    Commented Oct 25, 2018 at 9:31
  • $\begingroup$ If your bounding box contain 40 pixel from object and 60 from non object then TPR=40/100 and FPR=60/900 $\endgroup$
    – chetan gode
    Commented Dec 28, 2018 at 6:05
0
$\begingroup$

I think it is pretty late to answer your question. Correct me, if i am wrong.I am using Recall=$\frac{TP}{TP+FN}$.

I calculate IoU. If you impose threshold on IoU, it will be Recall. If the IoU is greater than threshold, then it will be TP else FN. You can do this for n frames in a sequence to get overall recall. Hope this helps.

I am just wondering how did you handle nans'. Did you consider them as FP?.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I'm sorry, I didn't understand your answer. Since the false positive rate is needed for the ROC curve, the question was how can one calculate TN. $\endgroup$
    – GoodDeeds
    Commented Oct 25, 2018 at 9:30
  • $\begingroup$ Also, what nans are you referring to? $\endgroup$
    – GoodDeeds
    Commented Oct 25, 2018 at 9:32
  • $\begingroup$ I am suggesting an alternative approach instead of calculating TN for fall out. You can calculate TP and FN to calculate Recall which later be used for ROC Curve. please check the links 1)stats.stackexchange.com/questions/7207/… 2) stats.stackexchange.com/questions/7207/… 3)en.wikipedia.org/wiki/Sensitivity_and_specificity(check the image in this link) $\endgroup$
    – sabarish amaravadi
    Commented Oct 25, 2018 at 9:36
  • $\begingroup$ sometimes my object tracking algorithm returns nan as IoU, if it could not detect an object. But the object is there. so, i am considering it as FP. you can just ignore it. It varies from algorithm to algorithm. $\endgroup$
    – sabarish amaravadi
    Commented Oct 25, 2018 at 9:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.