4
$\begingroup$

I have an object detection algorithm for which I would like to plot an ROC curve. For this, I need the values of the fall-out corresponding to values of recall.

The false positive rate, or fall-out, is defined as $$\text{Fall-out}=\frac{FP}{FP+TN}$$

In my data, a given image may have many objects. So, almost every image has at least one box. I am counting a predicted box as a true positive if its IOU with a truth box is above a certain threshold, and as a false positive otherwise. Any truth box with no prediction box having an IOU above a threshold counts as a false negative.

However, for the denominator, how is a "true negative" defined here? In the context of object detection, what does it mean to say that something is a true negative? Can any bounding box not detected, that does not correspond to an actual object (with an IOU above a certain threshold), be called a true negative? If so, wouldn't the number of true negatives be infinite?

From what I understand, $TN$ would be the number of "non-objects" that were not detected. How can this be quantified? Is it defined in some particular way, or undefined, or infinity? Would it even be possible to plot an ROC for this?

How can I do this?

Thank you.

$\endgroup$
  • 2
    $\begingroup$ Seems to me like false positive rate does not make sense in this scenario unless you can enumerate specific objects in your images that the algorithm should not detect. Otherwise, there are conceivably thousands of boxes which would be considered true negatives. $\endgroup$ – Joel May 2 '17 at 18:34
  • $\begingroup$ @Joel This was exactly what I thought. I think the number of true negatives could be infinite. Part of my question is to confirm if that is the case. Thank you for your comment. $\endgroup$ – GoodDeeds May 2 '17 at 18:36
  • 2
    $\begingroup$ Also, see this question $\endgroup$ – Joel May 2 '17 at 18:44
2
$\begingroup$

There are two errors that your algorithm can make. The first error is detecting an object when it's not there. This is the false positive. The second error is not detecting an object when it's there. This is the false negative.

To compute the false positive rate you want to compute how often it detected an object when the object was not there. This is the quotient of the false positives and all potential false positives: i.e.

$\frac{\text{detect box when no box}}{\text{all no box}}$

To compute the false negative rate you compute how often an object is not detected divided by how often the object could have been detected.

$\frac{\text{detect no box when box}}{\text{all box}}$

The key distinction between the false positive and false negative rate is which situation / hypothesis is true. False positives exist only when there are no boxes. False negatives exist only when there are boxes.

$\endgroup$
  • $\begingroup$ Thank you for your answer, Could you please elaborate on "False positives exist only when there are no boxes"? As my question asked how to calculate the number of "no boxes", do you mean to say that it is not possible, as the concept of a "no box" does not exist? $\endgroup$ – GoodDeeds May 1 '17 at 11:00
  • $\begingroup$ False positives / negatives are errors right? So let's say your algorithm tries to detect whether there is a box in n+m pictures. In n pictures there are no boxes. In m pictures there are boxes. False positives can only occur in pictures where there are no boxes so in n pictures: i.e. when can you make an error by detecting something that's not there? False negatives can only occur in pictures where there are boxes so in m pictures. $\endgroup$ – Toby May 1 '17 at 11:06
  • $\begingroup$ Thank you, I understood that statement now. Sorry, I realized that I have not been very clear, I am not sure if it has a different name, but in my case, a given image may have many objects. So, almost every image has at least one box. I am counting a predicted box as a true positive if its IOU with a truth box is above a certain threshold, and a false positive otherwise. Any truth box with no prediction box having an IOU above a threshold counts as a false negative. But what is a true negative? $\endgroup$ – GoodDeeds May 1 '17 at 11:10
  • $\begingroup$ I don't think I completely understand then. One way I would tackle the problem of counting boxes in pictures - this is the problem that you're attempting to tackle right? - is to compare the number of boxes in the picture to the number of boxes that the algorithm has found and use some criterion to minimize the deviation. But I don't understand what it is that you're trying to do here? $\endgroup$ – Toby May 1 '17 at 11:25
  • $\begingroup$ I am sorry for the lack of clarity. I have a set of images, and a program that detects all (possibly >1) objects, with their bounding boxes, in each of the images. I want to plot an ROC curve for this program. I think I can calculate the values of FP, FN, and TP, as mentioned in my previous comment. My question is, how do I calculate TN? From what I understand, TN would be the number of "non-objects" that were not detected. How can this be quantified? Is it defined in some particular way, or undefined, or infinity? Please let me know if I need to be clearer in explaining my problem. Thank you. $\endgroup$ – GoodDeeds May 1 '17 at 11:29
0
$\begingroup$

If there are 1000 pixels in an image out of that 100 belongs to an object then if your bounding box encloses all 100 pixels, then all other 900 pixels are true negative pixels

$\endgroup$
  • $\begingroup$ But wouldn't the hundred pixels count as one object, so contribute 1 unit to the TP? In your formulation how can you count any number of non-objects using the 900 pixels? $\endgroup$ – GoodDeeds Oct 25 '18 at 9:31
  • $\begingroup$ If your bounding box contain 40 pixel from object and 60 from non object then TPR=40/100 and FPR=60/900 $\endgroup$ – chetan gode Dec 28 '18 at 6:05
0
$\begingroup$

I think it is pretty late to answer your question. Correct me, if i am wrong.I am using Recall=$\frac{TP}{TP+FN}$.

I calculate IoU. If you impose threshold on IoU, it will be Recall. If the IoU is greater than threshold, then it will be TP else FN. You can do this for n frames in a sequence to get overall recall. Hope this helps.

I am just wondering how did you handle nans'. Did you consider them as FP?.

$\endgroup$
  • $\begingroup$ I'm sorry, I didn't understand your answer. Since the false positive rate is needed for the ROC curve, the question was how can one calculate TN. $\endgroup$ – GoodDeeds Oct 25 '18 at 9:30
  • $\begingroup$ Also, what nans are you referring to? $\endgroup$ – GoodDeeds Oct 25 '18 at 9:32
  • $\begingroup$ I am suggesting an alternative approach instead of calculating TN for fall out. You can calculate TP and FN to calculate Recall which later be used for ROC Curve. please check the links 1)stats.stackexchange.com/questions/7207/… 2) stats.stackexchange.com/questions/7207/… 3)en.wikipedia.org/wiki/Sensitivity_and_specificity(check the image in this link) $\endgroup$ – sabarish amaravadi Oct 25 '18 at 9:36
  • $\begingroup$ sometimes my object tracking algorithm returns nan as IoU, if it could not detect an object. But the object is there. so, i am considering it as FP. you can just ignore it. It varies from algorithm to algorithm. $\endgroup$ – sabarish amaravadi Oct 25 '18 at 9:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.