1
$\begingroup$

I am interested in reproducing the object detection results found in the whitepaper describing the YouTube BoundingBox dataset (https://arxiv.org/pdf/1702.00824.pdf). What I don't understand is how the authors can use the Mean Average Precision (mAP) metric to evaluate/train their network when using YouTube BoundingBox? As I understand it, precision is evaluated for object detection as:

precision = true_positives / (true_positives + false_positives)

While this makes a lot of sense for exhaustively labeled datasets like COCO, YouTube BoundingBox only tracks a single object over a few frames. To see an example of this, observe the zebra example in Figure 1 of the paper I link to above. While there are 5+ visible zebras in the frame, only one has a labeled bounding box. This means that even if the detection network was working exactly as a human would (observing and marking all of the zebras as zebras), it would have a low mAP as all of the non-labeled zebras would be considered as false positives.

Am I misunderstanding mAP, the training/evaluation method the paper used, or something else? If I am correct then it is pretty weird that Faster-RCNN can learn to recognize only the labeled object and ignore non-labeled objects. Perhaps this means that it learns the kind of objects that the human annotators were initially drawn to.

$\endgroup$

1 Answer 1

0
$\begingroup$

Basically, when calculating metrics, it pretends those unlabeled zebra no exist. (assuming COCO might also misses a 90% hidden zebra).

Eventually, the model might predict some unlabeled zebra as zebra, while the metrics think it's wrong. (which is not ideal) The hope is that, most frames are single object, hence labeled correctly. And the model, like human, learns to ignore some wrong knowledge.

$\endgroup$
1
  • 1
    $\begingroup$ But my point is that COCO is exhaustively labeled. That means that it would not miss the existing zebras in the frame, and therefore is correct in contrast to YouTube BoundingBox. The point about most frames containing a single object is well taken since that might mean that the model wouldn't be constantly presented with false-false-positives (incorrect false positives from unlabeled objects), but still the lack of exhaustive labeling fundamentally is a weakness in this dataset it sounds like. $\endgroup$
    – Kantthpel
    Jul 10, 2017 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.