59
$\begingroup$

In ImageNet classification papers top-1 and top-5 error rates are important units for measuring the success of some solutions, but what are those error rates?

In ImageNet Classification with Deep Convolutional Neural Networks by Krizhevsky et al. every solution based on one single CNN (page 7) has no top-5 error rates while the ones with 5 and 7 CNNs have (and also the error rate for 7 CNNs are better than for 5 CNNs).

Does this mean top-1 error rate is the best single error rate for one single CNN?

Is the top-5 error rate simply the accumulated error rate of five CNNs?

$\endgroup$
1
  • $\begingroup$ There is one more term that appears near these, i.e. "single crop". What is "single crop top-1 accuracy" seems to simply be the Accuracy, but can there be "single crop top-5 accuracy"? $\endgroup$ – Tomasz Gandor Jun 24 '20 at 6:23
73
$\begingroup$

[...] where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the mode.

First, you make a prediction using the CNN and obtain the predicted class multinomial distribution ($\sum p_{class} = 1$).

Now, in the case of top-1 score, you check if the top class (the one having the highest probability) is the same as the target label.

In the case of top-5 score, you check if the target label is one of your top 5 predictions (the 5 ones with the highest probabilities).

In both cases, the top score is computed as the number of times a predicted label matched the target label, divided by the number of data-points evaluated.

Finally, when 5-CNNs are used, you first average their predictions and follow the same procedure for calculating the top-1 and top-5 scores.

$\endgroup$
3
  • 1
    $\begingroup$ Why is this a valid metric? If the class isn't having the highest probability, what's the point? We can't use it in real world applications! $\endgroup$ – Black Jack 21 Jul 3 '20 at 6:12
  • $\begingroup$ One method may be to obtain the top-5 of differing models/networks, and re-weight based on multiple model scores, although this assumes sufficient resources are available to store several models, and there is enough time to load and test those models before you are required to make a decision $\endgroup$ – Tim Woolford Aug 9 '20 at 23:45
  • $\begingroup$ @BlackJack21 It is for a competition, and the point is to find classes with lower predicted probability which are the correct answer. $\endgroup$ – Volker Siegel Oct 22 '20 at 22:48
26
$\begingroup$

Your classifier gives you a probability for each class. Lets say we had only "cat", "dog", "house", "mouse" as classes (in this order). Then the classifier gives somehting like

0.1; 0.2; 0.0; 0.7

as a result. The Top-1 class is "mouse". The top-2 classes are {mouse, dog}. If the correct class was "dog", it would be counted as "correct" for the Top-2 accuracy, but as wrong for the Top-1 accuracy.

Hence, in a classification problem with $k$ possible classes, every classifier has 100% top-$k$ accuracy. The "normal" accuracy is top-1.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.