Machine learning classifiers often use the cross-entropy $\mathbb{H}[p,q]$, where $p$ is the true distribution (often a delta) and $q$ is the predicted distribution over classes (or can at least be interpreted that way).

Minimizing this is the same as minimizing the KL-divergence between the truth and the prediction, since $$ \mathbb{H}[p,q] = \mathcal{D}_\text{KL}[p||q] + \mathbb{H}[p] $$ where $\mathbb{H}[p]$ is the entropy of $p$ (zero for a delta, or constant wrt the model in any case).

Question: why don't we use $$ \mathcal{L}(p,q) = \mathbb{H}[p,q] + \mathbb{H}[q,p] = \mathcal{S}_\text{KL}[p,q] + \mathbb{H}[p] + \mathbb{H}[q] $$ where $\mathcal{S}_\text{KL}=\mathcal{D}_\text{KL}[p||q]+\mathcal{D}_\text{KL}[q||p]$ is a symmetric KL-divergence. Notice that this also tries to minimizes the uncertainty in the prediction, which seems like a reasonable thing to me.


Consider a classification context like you mentioned, where $q(y \mid x)$ is the model distribution over classes, given input $x$. $p(y \mid x)$ is the 'true' distribution, defined as a delta function centered over the true class for each data point:

$$p(y \mid x_i) = \left \{ \begin{array}{cl} 1 & y = y_i \\ 0 & \text{Otherwise} \\ \end{array} \right .$$

For the $i$th data point, the cross entropy $H(q, p)$ is:

$$H(q,p) = -\sum_y q(y \mid x_i) \log p(y \mid x_i)$$

Because $p(y \mid x_i) = 0$ when $y \ne y_i$, this requires summing over terms involving $\log(0)$, and $H(q,p)$ will be $-\infty$ or undefined.

  • $\begingroup$ Thanks for your answer! I guess it's ok to use $\mathbb{H}[p,q]$ because we always assume that $q(y|x)>0$? I guess it's reasonable in ML, but it seems a bit... strong. Though, could we not also just as reasonably let $p(y|x)=1-n\epsilon$ if $y=y_i$ and $\epsilon$ otherwise, where $n$ is the number of classes? $\endgroup$ – user3658307 Mar 6 '18 at 19:37
  • $\begingroup$ Yes, that's right...infinite values if q goes to 0 or 1. It's a well known issue with log loss (which is equivalent to cross entropy, as Ami Tavory mentioned). Regarding your workaround for $H(q,p)$, something like that would indeed avoid the issue $\endgroup$ – user20160 Mar 7 '18 at 0:42

For discrete $p$ and $q$, the loss is

$$ {\displaystyle H(p,q)=-\sum _{x}p(x)\,\log q(x).\!} H(p,q)=-\sum _{x}p(x)\,\log q(x).\! $$

This exactly corresponds to the expected loss under log-loss assumptions. Say you predict that the next symbol will be $x$ with probability $q(x)$, then your loss will be $- \log(q(x))$. The probability that this will happen, under the true probability $p$ is $p(x)$.

The log loss is very natural in cases such as online compression (where Arithmetic coding will use about the log of the inverse probability attributed to the symbol), or online gambling (where the log is the rate of doubling the capital); see here for example.

  • $\begingroup$ This seems to be answering why one would use $H(p,q)$ as a loss function, whereas the OP is asking why one would not use $H(p,q) + H(q,p)$ $\endgroup$ – user20160 Mar 6 '18 at 8:05
  • $\begingroup$ @user20160 I might be missing something (please let me know), but the gist of my answer is "because $H(p, q)$ is the expected loss (under log-loss), and therefore $H(p, q) + H(q, p)$ isn't. $\endgroup$ – Ami Tavory Mar 6 '18 at 8:12
  • $\begingroup$ Hmm, ok. I certainly agree with the merits of $H(p,q)$ for the reasons you mentioned, and I think it makes sense to point those out. The thing that's not clear to me is why that means another loss function shouldn't be used. $\endgroup$ – user20160 Mar 6 '18 at 8:35
  • $\begingroup$ @user20160 I think that, intuitively, it makes sense to optimize what you'll end up gaining/losing. There are secondary questions about this, e.g., should you take the expectation of this (or perhaps the worst case), is it robust, and so forth. In any case, I liked the specifics of you answer as well. $\endgroup$ – Ami Tavory Mar 6 '18 at 8:40
  • $\begingroup$ Thanks for your thoughts! I knew it was equivalent to the log-loss, but I am sort of wondering whether there is some relation to the log-loss for $\mathbb{H}[q,p]$ (since we are still minimizing the KL-divergence between the true distribution and the predictive one). It also just seems that minimizing the "symmetric" cross entropy still minimizes the log-loss, but additionally minimizes the predictive entropy. But perhaps the latter effect is not always desirable? $\endgroup$ – user3658307 Mar 6 '18 at 19:44

Cross-Entropy is one of the methods used to find how good is the predicted probability models.

The minimum value that the cross-entropy of ℍ[𝑝,𝑞] can have is when 𝑞=𝑝 which is ℍ[𝑝,𝑝], simple the entropy of the distribution 𝑝.

While evaluating different built models say 𝑞 and 𝑞', we often need to compare different them, and cross-entropy can be used here. The more the value is close to ℍ[𝑝,𝑝], the better is our model.

However, if we take symmetric cross entropy, though there is a lower bound here also, but it becomes difficult to compare two different models.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.