# Measuring the confidence of a softmax classification outcome

Suppose I have a softmax distribution produced by a classifier. There are four labels, and so the sum of the softmax probabilities over the four labels will be 1.0.

I am looking for a measurement of how "certain" or "confident" the classifier's prediction is.

For example, suppose for two data instances, the softmax distributions are:

outcome1 = [0.25, 0.25, 0.20, 0.30]
outcome2 = [0.02, 0.94, 0.02, 0.02]


It's clear that the classifier is more confident in outcome2 since there is a large probability mass (0.94) on one of the labels. On the other hand, the classifier is less confident in outcome1 since the probabilities are fairly equal.

So I'm looking for a way to quantify this degree of "certainty" in a classifier's prediction.

One thing I was thinking of was computing the Shannon entropy of each outcome:

from scipy.stats import entropy
print(entropy(outcome1, base=2)) # 1.9854752972273344
print(entropy(outcome2, base=2)) # 0.4225426691977457


Can I say that the classifier is  1.98 / 0.42 = 4.7 times more confident in outcome2 versus outcome1?

• Thanks for the answer. I was unaware that there were formal definitions of certainty and confidence. Given your explanation of certainty (that lead to actually correct outcomes), I suppose I can measure that with accuracy, F1, etc. I'm less conversant with confidence. I'll take a look at that paper you mentioned. Dec 31, 2020 at 19:47