3
$\begingroup$

Usually the last layer in multiclass classification models is a softmax, which is essentially a vector with elements the confidences for each class. The standard top-1 accuracy takes account only if the class with the highest confidence matches the true class.

However, the confidence distribution of the softmax output can give additional information about the probability of an input sample belonging to a certain class.

Taking this into account, what other metrics can be used to estimate the model's accuracy more realistically? For example, entropy can be very informative about the softmax distribution. What other metrics which consider the softmax spikiness (how peaky or diffuse the output probabilities are) can be useful for determining the model's real accuracy?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

I think, one possible way to is to extend the definition of accuracy is by using the soft-labels, i.e. the probabilities. Normally, we have a one-hot vector indicating the predicted class, e.g. $\hat y=[0,1,0,0]$, and the ground truth, e.g. $y=[0,0,1,0]$. What we do in accuracy calculation is to take the dot product of these, i.e. $\hat y^T y$, and accumulate for all the test samples. This doesn't account for the confidence level as you mentioned.

If you were to use the predicted probabilities $\hat p = [p_1, p_2, p_3, p_4]$ instead of one-hot decisions, the dot-product $\hat p^Ty$ would be $p_j$ where the sample belongs to class $j$. So, the alternative accuracy would be the sum of the predicted probabilities for the ground truth class. This way, even in wrongly predicted samples, we could add something $>0$ and for the correctly predicted samples, we'll most probably add something $<1$.

This seems to be a straightforward and simple extension, but I think it's quite intuitive as well. For example, for a given test sample, if the predicted probability for the ground truth class is $0.6$, it's like when we see this sample in the decision process, there is $60 \%$ chance that we'll predict it correctly, and $40\%$ chance that we won't, i.e. if we were to see the same sample for $100$ times, and assuming independent decisions, we would predict it correctly $60$ times and wrongly $40$ times. That means a $0.6$ accuracy for that sample and its contribution to the overall test set would be $0.6$, instead of $1$ as before.

Improve this answer
$\endgroup$
6
  • 2
    $\begingroup$ The name for this is the expected (classification) accuracy. $\endgroup$
    – Arya McCarthy
    Commented Apr 2, 2021 at 20:46
  • $\begingroup$ @AryaMcCarthy yeah, I didn't explicitly write that name since I haven't surprisingly encountered with the formal definition. $\endgroup$
    – gunes
    Commented Apr 2, 2021 at 20:48
  • $\begingroup$ Very interesting approach. Does the model need to be calibrated (outputing true probabilities instead of confidences) to use this metric? $\endgroup$
    – Andrew
    Commented Apr 3, 2021 at 10:40
  • $\begingroup$ Also do you think other metrics such as AUC or cross entropy could be useful? $\endgroup$
    – Andrew
    Commented Apr 3, 2021 at 11:29
  • 1
    $\begingroup$ Since this is like calculating some sort of expected value, calibrated probabilities reflect more realistic estimates. I think cross entropy can also be useful to evaluate the uncertainty, e.g. two models having the same accuracy but lower cross entropy loss, it might indicate more certain decision. I'm not sure how to generalise AUC for multi-class. $\endgroup$
    – gunes
    Commented Apr 3, 2021 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.