Why am I getting information entropy greater than 1?

Question

I implemented the following function to calculate entropy:

from math import log

def calc_entropy(probs):
    my_sum = 0
    for p in probs:
        if p > 0:
            my_sum += p * log(p, 2)

    return - my_sum

Result:

>>> calc_entropy([1/7.0, 1/7.0, 5/7.0])
1.1488348542809168
>>> from scipy.stats import entropy # using a built-in package 
                                    # give the same answer
>>> entropy([1/7.0, 1/7.0, 5/7.0], base=2)
1.1488348542809166

My understanding was that entropy is between 0 and 1, 0 meaning very certain, and 1 meaning very uncertain. Why do I get measure of entropy greater than 1?

I know that if I increase size of log base, the entropy measure will be smaller, but I thought base 2 was standard, so I don't think that's the problem.

I must be missing something obvious, but what?

Answer 1

Entropy is not the same as probability.

Entropy measures the "information" or "uncertainty" of a random variable. When you are using base 2, it is measured in bits; and there can be more than one bit of information in a variable.

In this example, one sample "contains" about 1.15 bits of information. In other words, if you were able to compress a series of samples perfectly, you would need that many bits per sample, on average.

Answer 2

The maximum value of entropy is $\log k$, where $k$ is the number of categories you are using. Its numeric value will naturally depend on the base of logarithms you are using.

Using base 2 logarithms as an example, as in the question: $\log_2 1$ is $0$ and $\log_2 2$ is $1$, so a result greater than $1$ is definitely wrong if the number of categories is $1$ or $2$. A value greater than $1$ will be wrong if it exceeds $\log_2 k$.

In view of this it is fairly common to scale entropy by $\log k$, so that results then do fall between $0$ and $1$,

Answer 3

Earlier answers, specifically: "Entropy is not the same as probability." and "the maximum value of entropy is log 𝑘" are both correct.

As stated earlier "Entropy measures the "information" or "uncertainty" of a random variable." Information can be measured in bits and when doing so log2 should be used. However, if a different information unit is used, the amount of information changes simply because the unit can encode more information. As an example, 1 bit can encode two events 0,1, while 1 ban can encode 10 different events, it follows then that 1 ban = 3.322 bits (3 bits = 8 events).

In summary, using entropy values between 0-1 and >1 is really no different as long as you use the same entropy units across comparisons. However, for some applications (Cross-entropy loss) using a value between 0 and 1 may be more convenient.