Suppose I have a softmax distribution produced by a classifier. There are four labels, and so the sum of the softmax probabilities over the four labels will be 1.0.
I am looking for a measurement of how "certain" or "confident" the classifier's prediction is.
For example, suppose for two data instances, the softmax distributions are:
outcome1 = [0.25, 0.25, 0.20, 0.30]
outcome2 = [0.02, 0.94, 0.02, 0.02]
It's clear that the classifier is more confident in outcome2
since there is a large probability mass (0.94) on one of the labels. On the other hand, the classifier is less confident in outcome1
since the probabilities are fairly equal.
So I'm looking for a way to quantify this degree of "certainty" in a classifier's prediction.
One thing I was thinking of was computing the Shannon entropy of each outcome:
from scipy.stats import entropy
print(entropy(outcome1, base=2)) # 1.9854752972273344
print(entropy(outcome2, base=2)) # 0.4225426691977457
Can I say that the classifier is 1.98 / 0.42 = 4.7
times more confident in outcome2
versus outcome1
?