Intuitive or quantitative explanation of why we care about mean average precision (mAP) for CNN classifiers?

Question

Consider CNN classifiers applied to some image classification tasks: to fix ideas, let's consider the ImageNet Challenge, where each image belongs to 1 of 1000 nonoverlapping classes, even though the question is more general.

Usually, when people summarize the progress in time of CNN performance on ImageNet, the mAP metric (mean Average Precision) is reported. Given the precision-recall curve of the classifier for a specific class, $AP$ (Average Precision) is usually defined as the average of precision $P(R)$ over $N=11$ equally spaced recall values $R_1=0, R_2=0.1,\dots,R_N=1.0$:

$$ AP =\frac{1}{N}\sum_{i=1}^{N} P(R_i)$$

mAP is then defined as the average of $AP$ over all classes (e.g., 1000 classes for the ImageNet Challenge).

I get the reason for the average over all classes (i.e., the "m" in "mAP"), but I don't see the need for averaging over the precision-recall curve. The precision-recall curve is used to choose a classification threshold for a classifier. However, this doesn't resemble the way CNN classifiers are used in practice. For each input image, a CNN classifier outputs 1000 numbers in $[0,1]$ (one for each class), and it classifies the image to the class for which this number is maximum¹. It doesn't actually use a threshold at all. And this is the way they're used in practice: nowadays you can even download apps on iOS/Android and see that they classify an object to the maximum probability class.

So, why don't just report the average proportion of correctly classified cases over all classes? I think this has to do with the fact that "accuracy is an improper scoring rule". I don't know exactly what it means, but again, I've seen this sentence used in the context of setting a threshold for a classifer. a CNN classifies without any threshold, so why do we need "a proper scoring rule" (whatever that means) to study the progress of CNN classifiers over time?

---

¹_{I'm not calling these numbers "estimated probabilities" because it's pretty well known that these estimates are not calibrated at all for modern CNN architectures, unlike for old shallow MLPs. Thus interpreting them as probabilities is highly questionable.}

Answer 1

I believe the reasons are practical and the main one for preferring mAP to accuracy is class imbalance. By evaluating against mAP, to perform well the classifier needs to be able to handle the smaller classes. In this respect, I think the justification is similar to why average precision is used in information retrieval and NLP.

In the ILSVRC classification and subsequent localisation/detection challenges from 2010 to 2017 the main metric for the contest was top-k accuracy. Top-k was selected because contest organizers did not want to penalise algorithms for detecting objects which were present in the scene, but were not labelled in the ground truth. The instructions for ILSVRC make reference to the test data being carefully selected for various relevant properties and I would assume that class balance was one of those.

People care about mAP in the context of object detection not image classification generally. Why mAP? A pragmatic reason was that it was popularised by PascalVOC. The PascalVOC people gave a few justifications for their choice of evaluation metrics which are informed by the dataset they created:

1. there can be multiple instances of multiple classes in each image
2. the number of instances of each class wasn't balanced
3. they wanted the metrics to be algorithm independent
4. AUC didn't differentiate well between the competitive algorithms in the contest

There's further elaboration in PascalVOC paper:

Everingham, Mark, et al. "The pascal visual object classes (voc) challenge." International journal of computer vision 88.2 (2010): 303-338.

Personally, I think mAP makes sense because you want to identify multiple objects in an image, and it's hard to avoid class imbalance in object detection datasets once you start counting object instances. Photos of people tend to have more than one person but photos of hairdryers tend to only have a single hair dryer. This is partly due to bias in the underlying social media and photoshare websites from which these datasets are drawn. See the following diagram showing the object instance distributions and imbalance in MSCOCO and PascalVOC:

(Source)

There is a similar issue in semantic segmentation because the pixel-wise class distributions are frequently quite skewed. Hence why the preferred metrics there are usually either the Dice score or the Jaccard coefficient which are close related to the F1 score.