When computing mAP for an object detection model, how many detections should one consider?

Question

I am trying to write some code to evaluate the MS COCO style mAP (mean average precision, average computed at the category level) at different IOU levels in the context of object detection with a Faster-RCNN type model. Fixing a given category, here is how I understand the pseudocode in order to get the recall, precision tuples assuming I have a dict as in python with the structure initialized as:

eval[category][c_val][iou_val] = {"true_positives":0} # for each iou value
eval[category]["num_detections] = 0

Pseudocode:

Set a max number of detections N
for each test image
  for each detection in order of confidence value (c_val):
    get the ground truths of the same category for this image
    get the ground truth with the largest IOU, call it bb with iou:=max_iou
    add a new confidence value in the dict if c_val is not already a key (see **)
    for confidence value c_val_ less than or equal to c_val
    eval[category][c_val_]["num_detections"] += 1
    remove bb from the ground truth list to prevent multiple detections
    for iou_val in [0.50, 0.55, 0.60, 0.65, 0.7, 0.75, 0.80, 0.85, 0.9, 0.95]
       if max_iou is >= iou_val
            for each c_val in the eval dict less than or equal to c_val
               eval[category][c_val][iou_val]["true_positives"] += 1
      

** (with the same dict entry initialized as the confidence score immediately below it in the dict if one exists)

A Faster R-CNN model can output as many detections as there are region proposals, which is a tuned hyperparameter (assume that non-maximum suppresion has been applied both for the region proposal network output and for detection output). For instance suppose an image has 5 boxes that are detected separately with high IOU and high confidence by a faster R-CNN model with num_region_proposals=300, and the remaining 295 boxes with very low IOU and very low confidence values. If you include all of these in the computation of the mean average precision then this will lower your score dramatically by increasing the number of false positives with some increase in recall.

On the other hand, you may have the scenario where for an image with a lot of objects you have quite a few low confidence value predictions with high IOU at the "back of the list" of the confidence-ordered detections.

My question is, with this approach, how many detections should I consider per image both to compute mAP? I am imagining in practice (not for evaluating metrics, but for actual use of the model) you would simply set a confidence threshold, but I'm not sure if this is correct either.

For instance there might be a min number of detections even if the confidence threshold isn't met so that at least you have some kind of output since a model with low confidence scores could end up rarely outputting anything at all, which is obviously useless. Any insights appreciated.

Edit: Just thinking about this some more it seems there are some ideas for how to approach this:

1. a fixed number of detections
2. a confidence value threshold
3. a min number of detections + a confidence value threshold
4. setting the number of detections to equal the number of objects in the image (seems like cheating kinda)
5. some sort of general heuristic based on the confidence values returned (i.e. if there are a lot of values in the range (0.45, 0.49999...), maybe you shouldn't be too hasty to cut off strictly at 0.5)
6. as suggested in the comments, a top-k prediction for the k highest confidence values

Answer 1

To give an answer to my own question, I think there are probably many approaches in practice, and this is part of the "model" in question. For example, I recently trained a Faster R-CNN Feature Pyramid Network on a dataset. During evaluation I set the min confidence value to be $0.6$, and the NMS for detected boxes to be $0.7$ with a max detection of $100$ boxes, and this yielded a decent mean average precision on the test set (better than a looser 0.5 confidence threshold).

Really there are two scenarios:

1. You have access to all the test data set and can tweak the parameters to get the best outcome (which some people might argue is "cheating" since you are using test performance to guide model parameters, but I'm sure a lot of people do this anyway...)

2. You don't have access to the some/all test data a priori, maybe a model used in production. In this case you have to come up with a strategy and hope it works, where the default from what I can tell seems to be just setting a confidence threshold and a max number of detections.