Metrics for multiclass classification model accuracy Usually the last layer in multiclass classification models is a softmax, which is essentially a vector with elements the confidences for each class. The standard top-1 accuracy takes account only if the class with the highest confidence matches the true class.
However, the confidence distribution of the softmax output can give additional information about the probability of an input sample belonging to a certain class.
Taking this into account, what other metrics can be used to estimate the model's accuracy more realistically? For example, entropy can be very informative about the softmax distribution. What other metrics which consider the softmax spikiness (how peaky or diffuse the output probabilities are) can be useful for determining the model's real accuracy?
 A: I think, one possible way to is to extend the definition of accuracy is by using the soft-labels, i.e. the probabilities. Normally, we have a one-hot vector indicating the predicted class, e.g. $\hat y=[0,1,0,0]$, and the ground truth, e.g. $y=[0,0,1,0]$. What we do in accuracy calculation is to take the dot product of these, i.e. $\hat y^T y$, and accumulate for all the test samples. This doesn't account for the confidence level as you mentioned.
If you were to use the predicted probabilities $\hat p = [p_1, p_2, p_3, p_4]$ instead of one-hot decisions, the dot-product $\hat p^Ty$ would be $p_j$ where the sample belongs to class $j$. So, the alternative accuracy would be the sum of the predicted probabilities for the ground truth class. This way, even in wrongly predicted samples, we could add something $>0$ and for the correctly predicted samples, we'll most probably add something $<1$.
This seems to be a straightforward and simple extension, but I think it's quite intuitive as well. For example, for a given test sample, if the predicted probability for the ground truth class is $0.6$, it's like when we see this sample in the decision process, there is $60 \%$ chance that we'll predict it correctly, and $40\%$ chance that we won't, i.e. if we were to see the same sample for $100$ times, and assuming independent decisions, we would predict it correctly $60$ times and wrongly $40$ times. That means a $0.6$ accuracy for that sample and its contribution to the overall test set would be $0.6$, instead of $1$ as before.
