Why don't we use a symmetric cross-entropy loss?

Question

Machine learning classifiers often use the cross-entropy $\mathbb{H}[p,q]$, where $p$ is the true distribution (often a delta) and $q$ is the predicted distribution over classes (or can at least be interpreted that way).

Minimizing this is the same as minimizing the KL-divergence between the truth and the prediction, since $$ \mathbb{H}[p,q] = \mathcal{D}_\text{KL}[p||q] + \mathbb{H}[p] $$ where $\mathbb{H}[p]$ is the entropy of $p$ (zero for a delta, or constant wrt the model in any case).

Question: why don't we use $$ \mathcal{L}(p,q) = \mathbb{H}[p,q] + \mathbb{H}[q,p] = \mathcal{S}_\text{KL}[p,q] + \mathbb{H}[p] + \mathbb{H}[q] $$ where $\mathcal{S}_\text{KL}=\mathcal{D}_\text{KL}[p||q]+\mathcal{D}_\text{KL}[q||p]$ is a symmetric KL-divergence. Notice that this also tries to minimizes the uncertainty in the prediction, which seems like a reasonable thing to me.

Answer 1

Consider a classification context like you mentioned, where $q(y \mid x)$ is the model distribution over classes, given input $x$. $p(y \mid x)$ is the 'true' distribution, defined as a delta function centered over the true class for each data point:

$$p(y \mid x_i) = \left \{ \begin{array}{cl} 1 & y = y_i \\ 0 & \text{Otherwise} \\ \end{array} \right .$$

For the $i$th data point, the cross entropy $H(q, p)$ is:

$$H(q,p) = -\sum_y q(y \mid x_i) \log p(y \mid x_i)$$

Because $p(y \mid x_i) = 0$ when $y \ne y_i$, this requires summing over terms involving $\log(0)$, and $H(q,p)$ will be $-\infty$ or undefined.

Answer 2

For discrete $p$ and $q$, the loss is

$$ {\displaystyle H(p,q)=-\sum _{x}p(x)\,\log q(x).\!} H(p,q)=-\sum _{x}p(x)\,\log q(x).\! $$

This exactly corresponds to the expected loss under log-loss assumptions. Say you predict that the next symbol will be $x$ with probability $q(x)$, then your loss will be $- \log(q(x))$. The probability that this will happen, under the true probability $p$ is $p(x)$.

The log loss is very natural in cases such as online compression (where Arithmetic coding will use about the log of the inverse probability attributed to the symbol), or online gambling (where the log is the rate of doubling the capital); see here for example.

Answer 3

Cross-Entropy is one of the methods used to find how good is the predicted probability models.

The minimum value that the cross-entropy of ℍ[𝑝,𝑞] can have is when 𝑞=𝑝 which is ℍ[𝑝,𝑝], simple the entropy of the distribution 𝑝.

While evaluating different built models say 𝑞 and 𝑞', we often need to compare different them, and cross-entropy can be used here. The more the value is close to ℍ[𝑝,𝑝], the better is our model.

However, if we take symmetric cross entropy, though there is a lower bound here also, but it becomes difficult to compare two different models.

http://www.cs.rochester.edu/u/james/CSC248/Lec6.pdf