Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
If I understand right, general cross-entropy cost function can be written as:
$$c := - \sum_{i} t_{i} \log (a_i)$$
where vector $\mathbf{t}$ is 'true' discrete pdf and the vector $\mathbf{a}$ is the predicted pdf for the current input. Is it easily provable that $\mathbf{t} \equiv \mathbf{a}$ minimize the cost?
Obviously this is the case when $\mathbf{t}$ is all 0s except one 1, the usual case where we are sure which category the current input sample belongs to.
Gibb's Inequality states that for two vectors of probabilities $t \in [0, 1]^n$ and $a \in [0, 1]^n$, we have $$ -\sum_{i=1}^n t_i \log(t_i) \le -\sum_{i=1}^n t_i \log(a_i) $$ with equality if and only if $t = a$, and hence the cross-entropy cost function is minimized when $t = a$. The proof is simple, and is found on the Wikipedia page.
