One of the evaluation method for ImageNet Competition (classify 1,000 categories images) is top-5 error, what does that mean?
$\begingroup$
$\endgroup$
0
Franck Dernoncourt
44.8k32 gold badges166 silver badges282 bronze badges
asked Dec 16, 2014 at 23:59
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
3
Top-5 error, also known as rank-5 error is simply an instantiation of Rank-N error metric with $(N=5)$.
Rank $N$ error is the fraction of test samples $x_i$ where the correct label $y_i$ does not appear in the top $N$ predicted results of the model when results are sorted in decreasing order of confidence, or $P(y_i|x_i)$.
In ILSVRC 2014, the error metric for classification was:
\[ e = \tfrac{1}{n} \cdot \sum_k \min_{i} d(c_i, C_k) \]
where
\[ d(a, b) = \begin{cases} 0 & \text{if } a = b \\ 1 & \text{otherwise} \end{cases} \]
$c_i$ is the predicted label and $C_k$ is the ground truth label. There are $n$ possible labels with $k=1, ..., n$.
-
$\begingroup$ Great! One clarification, what does c and capitalized C mean? $\endgroup$ Jan 8, 2015 at 18:14
-
$\begingroup$ Here big C is one of the ground truth labels and little c is one of the predicted labels. $\endgroup$– sabalabaJan 8, 2015 at 19:54
-
1$\begingroup$ sabalaba -- could you add that information to your answer? $\endgroup$– Glen_bJan 1, 2016 at 0:40
$\begingroup$
$\endgroup$
the top-5 error rate (of a model) is the fraction of test images for which the correct label is not among the five labels considered most probable by the model [1]
$\begingroup$
$\endgroup$
I think the conditional evaluation of $d(A_i, B_k)$ should read: $d(Ai, Bk) = 0$, if $A_i$ occurs in the top N probable classifications for a given ground truth class Bk. = 1, if it does not.
Here k takes values 0 to N-1. Where N is the number of classes. For each $B_k$, there are $A_i$ probable classifications, where i takes values [0, N-1]
