I implemented the following function to calculate entropy:
from math import log def calc_entropy(probs): my_sum = 0 for p in probs: if p > 0: my_sum += p * log(p, 2) return - my_sum
>>> calc_entropy([1/7.0, 1/7.0, 5/7.0]) 1.1488348542809168 >>> from scipy.stats import entropy # using a built-in package # give the same answer >>> entropy([1/7.0, 1/7.0, 5/7.0], base=2) 1.1488348542809166
My understanding was that entropy is between 0 and 1, 0 meaning very certain, and 1 meaning very uncertain. Why do I get measure of entropy greater than 1?
I know that if I increase size of log base, the entropy measure will be smaller, but I thought base 2 was standard, so I don't think that's the problem.
I must be missing something obvious, but what?