91
$\begingroup$

In ImageNet classification papers top-1 and top-5 error rates are important units for measuring the success of some solutions, but what are those error rates?

In ImageNet Classification with Deep Convolutional Neural Networks by Krizhevsky et al. every solution based on one single CNN (page 7) has no top-5 error rates while the ones with 5 and 7 CNNs have (and also the error rate for 7 CNNs are better than for 5 CNNs).

Does this mean top-1 error rate is the best single error rate for one single CNN?

Is the top-5 error rate simply the accumulated error rate of five CNNs?

Improve this question
$\endgroup$
1
  • 3
    $\begingroup$ There is one more term that appears near these, i.e. "single crop". What is "single crop top-1 accuracy" seems to simply be the Accuracy, but can there be "single crop top-5 accuracy"? $\endgroup$
    – Tomasz Gandor
    Commented Jun 24, 2020 at 6:23

2 Answers 2

Reset to default
103
$\begingroup$

[...] where the top-5 error rate is the fraction of test images for which the correct label is not among the five labels considered most probable by the mode.

First, you make a prediction using the CNN and obtain the predicted class multinomial distribution ($\sum p_{class} = 1$).

Now, in the case of the top-1 score, you check if the top class (the one with the highest probability) is the same as the target label.

In the case of the top-5 score, you check if the target label is one of your top 5 predictions (the 5 ones with the highest probabilities).

In both cases, the top score is computed as the number of times a predicted label matched the target label, divided by the number of data points evaluated.

Finally, when 5-CNNs are used, you first average their predictions and follow the same procedure for calculating the top-1 and top-5 scores.

Improve this answer
$\endgroup$
5
  • 5
    $\begingroup$ Why is this a valid metric? If the class isn't having the highest probability, what's the point? We can't use it in real world applications! $\endgroup$
    – Black Jack 21
    Commented Jul 3, 2020 at 6:12
  • $\begingroup$ One method may be to obtain the top-5 of differing models/networks, and re-weight based on multiple model scores, although this assumes sufficient resources are available to store several models, and there is enough time to load and test those models before you are required to make a decision $\endgroup$
    – Tim Woolford
    Commented Aug 9, 2020 at 23:45
  • $\begingroup$ @BlackJack21 It is for a competition, and the point is to find classes with lower predicted probability which are the correct answer. $\endgroup$
    – Volker Siegel
    Commented Oct 22, 2020 at 22:48
  • 1
    $\begingroup$ But it specifically states "top-1 error". What does a 40% "top-1 error" mean? 60% accuracy or 40% accuracy? ( It also says "validation error rates") $\endgroup$
    – Bersan
    Commented Apr 4, 2021 at 14:44
  • $\begingroup$ @Bersan 40% top-1 error should mean 60% accuracy. $\endgroup$
    – Ganesh Tata
    Commented May 10, 2022 at 21:31
42
$\begingroup$

Your classifier gives you a probability for each class. Lets say we had only "cat", "dog", "house", "mouse" as classes (in this order). Then the classifier gives somehting like

0.1; 0.2; 0.0; 0.7

as a result. The Top-1 class is "mouse". The top-2 classes are {mouse, dog}. If the correct class was "dog", it would be counted as "correct" for the Top-2 accuracy, but as wrong for the Top-1 accuracy.

Hence, in a classification problem with $k$ possible classes, every classifier has 100% top-$k$ accuracy. The "normal" accuracy is top-1.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.