If we have a model which outputs class probabilities for $K$ classes, e.g. a NN with softmax layer, how can we return an aggregate "confidence metric"?

Some intuitive ideas would be the probability of the highest class or the difference in probability between the top 2 classes.

Is there a standard choice here? If not, what are the options.

  • $\begingroup$ What does the value of the loss function (probably categorical crossentropy) not give you? $\endgroup$
    – Dave
    Oct 30, 2020 at 3:00
  • $\begingroup$ Well the cross entropy in most contexts would just be $-log(p_k)$ where $p_k$ is the probability class $k$ with highest probability. So this only takes into account the max value. However, we may want to consider the difference between top 2 values for example which is useful. $\endgroup$ Oct 30, 2020 at 3:06
  • $\begingroup$ Perhaps you can say more about what a confidence score is. Crossentropy loss is a strictly proper scoring rule that aims to find the correct probabilities, so if you want to measure if your model is close to modeling the phenomenon under consideration, that’s a pretty good measure. $\endgroup$
    – Dave
    Oct 30, 2020 at 3:18


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.