Why is the cross entropy of the same probability distribution not 0?

Question

From what I've been reading, if there is no underlying difference between the 2 probabilities distributions we would have perfect entropy.

I'm putting an example below. Can anybody explain why the cross entropy of two exactly equal probability distributions is not 0 here?

### example of calculating cross entropy for identical distributions
from math import log2

### calculate cross-entropy
def cross_entropy(p, q):
    return -sum([p[i]*log2(q[i]) for i in range(len(p))])

### define data
p = [0.10, 0.40, 0.50]
# calculate cross entropy H(P, P)
ce_pp = cross_entropy(p, p)

print('H(P, P): %.3f bits' % ce_pp)

Result = 1.36

Answer 1

We can see that your understanding is not correct by inspecting the definition of cross-entropy: $$H(p,q)=-\sum_i p_i \log q_i .$$

• The function $f(q) = -\log(q)$ is zero only for $q=1$. It's sufficient to observe that the function is monotonic and therefore has at most 1 zero.

• The function $g(p) = p$ is zero only for $p=0$.

So we know that to achieve $H(p,q)=0$, we require some configuration of $p_i=0$ or $q_i=1$. The example in OP's post doesn't have any $p_i=0$ or $q_i=1$, so the claim in the question is false.

An example of $p,q$ that have $H(p,q)=0$ is $p = [0, 1]$ and $q=[0,1]$.

(Note that not every choice that has some $p_i=0$ or $q_i=1$ will have $H(p,q)=0$.)