If I understand right, general cross-entropy cost function can be written as:

$$c := - \sum_{i} t_{i} \log (a_i)$$

where vector $\mathbf{t}$ is 'true' discrete pdf and the vector $\mathbf{a}$ is the predicted pdf for the current input. Is it easily provable that $\mathbf{t} \equiv \mathbf{a}$ minimize the cost?

Obviously this is the case when $\mathbf{t}$ is all 0s except one 1, the usual case where we are sure which category the current input sample belongs to.


1 Answer 1


Gibb's Inequality states that for two vectors of probabilities $t \in [0, 1]^n$ and $a \in [0, 1]^n$, we have $$ -\sum_{i=1}^n t_i \log(t_i) \le -\sum_{i=1}^n t_i \log(a_i) $$ with equality if and only if $t = a$, and hence the cross-entropy cost function is minimized when $t = a$. The proof is simple, and is found on the Wikipedia page.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.