I can understand why cross-entropy works as a cost function, if we classify a sample as belonging to one of the classes $c_1, ..., c_n$:
- $P_{true}(c_i = c_i^{true}) = 1$;
- $H_e = -\log{P_{predicted}(c_i = c_i^{true})}$, because all other true probabilities are zeroes.
Now, if you predict probability close to $1.0$, cross-entropy aims toward zero. If you predict small probability for a true class, than you are getting penalized a lot by logarithm.
But what if you don't know a true label. The only you know is that:
- $P_{true}(c_1) = 0.3$
- $P_{true}(c_2) = 0.25$
- $P_{true}(c_3) = 0.45$
Now if:
- $P_{pred}(c_1) = 0.3$
- $P_{pred}(c_2) = 0.25$
- $P_{pred}(c_3) = 0.45$
and we compute cross-entropy:
- $H_e = -(0.3 \log{0.3} + 0.25 \log{0.25} + 0.45 \log{0.45}) / 3 \approx 0.356$
for a completelly true probabilistic prediction. Why is so?