# Cross-entropy for probabilistic classes

I can understand why cross-entropy works as a cost function, if we classify a sample as belonging to one of the classes $c_1, ..., c_n$:

• $P_{true}(c_i = c_i^{true}) = 1$;
• $H_e = -\log{P_{predicted}(c_i = c_i^{true})}$, because all other true probabilities are zeroes.

Now, if you predict probability close to $1.0$, cross-entropy aims toward zero. If you predict small probability for a true class, than you are getting penalized a lot by logarithm.

But what if you don't know a true label. The only you know is that:

• $P_{true}(c_1) = 0.3$
• $P_{true}(c_2) = 0.25$
• $P_{true}(c_3) = 0.45$

Now if:

• $P_{pred}(c_1) = 0.3$
• $P_{pred}(c_2) = 0.25$
• $P_{pred}(c_3) = 0.45$

and we compute cross-entropy:

• $H_e = -(0.3 \log{0.3} + 0.25 \log{0.25} + 0.45 \log{0.45}) / 3 \approx 0.356$

for a completelly true probabilistic prediction. Why is so?

$H_{wrong} = -(.5 \log .25 + .5 \log .75) \approx .837$
$H_{correct} = -(.5 \log .5 + .5 \log .5) \approx .693$