Suppose I have a softmax distribution produced by a classifier. There are four labels, and so the sum of the softmax probabilities over the four labels will be 1.0.
I am looking for a measurement of how "certain" or "confident" the classifier's prediction is.
For example, suppose for two data instances, the softmax distributions are:
outcome1 = [0.25, 0.25, 0.20, 0.30] outcome2 = [0.02, 0.94, 0.02, 0.02]
It's clear that the classifier is more confident in
outcome2 since there is a large probability mass (0.94) on one of the labels. On the other hand, the classifier is less confident in
outcome1 since the probabilities are fairly equal.
So I'm looking for a way to quantify this degree of "certainty" in a classifier's prediction.
One thing I was thinking of was computing the Shannon entropy of each outcome:
from scipy.stats import entropy print(entropy(outcome1, base=2)) # 1.9854752972273344 print(entropy(outcome2, base=2)) # 0.4225426691977457
Can I say that the classifier is
1.98 / 0.42 = 4.7 times more confident in