Is relative entropy equal to cross-entropy during optimization?

Question

I came across a saying that estimates of KL divergence, otherwise known as relative entropy, of the truth of a random variable and its prediction ($y$ and $\hat{y}$) is equal to their cross entropy because entropy + KL divergence = cross entropy or something similar, and that one of the three elements of the formula is fixed or negligible somehow and therefore falls out during estimation or optimization.

And because they are equal, optimizing relative entropy is just the same as using a cross-entropy loss function, which is the more common method. Is this true? or is there a specific application or condition for this to hold?

Answer 1

In many situations, we seek to approximate a 'true' or target distribution $p$ with a model or approximation $q_\theta$ (parameterized by $\theta$). To do this, we search for parameters that minimize a chosen measure of dissimilarity between $p$ and $q_\theta$. The KL divergence is a common choice. It can always be expressed as the difference between a cross entropy term and an entropy term. For two distributions $p_1$ and $p_2$:

$$D_{KL}(p_1 \parallel p_2) = H(p_1, p_2) - H(p_1)$$

The entropy term can always be ignored if we're minimizing the 'forward' KL divergence $D_{KL}(p \parallel q_\theta)$, but cannot be ignored if we're minimizing the 'reverse' KL divergence $D_{KL}(q_\theta \parallel p)$

Forward KL divergence

When minimizing the forward KL divergence, we seek the optimal parameters $\theta^*$ as follows:

$$\theta^* = \arg \min_\theta \ D_{KL}(p \parallel q_\theta)$$ $$= \arg \min_\theta \ H(p, q_\theta) - H(p)$$

Note that the entropy $H(p)$ doesn't depend on $\theta$; it's just an additive constant and can be ignored for the purpose of optimization. That is, the parameters that minimize the forward KL divergence are the same as those that minimize the cross entropy $H(p, q_\theta)$:

$$\theta^* = \arg \min_\theta \ H(p, q_\theta)$$

Maximum likelihood estimation is a particularly common instance of this problem. In this case, $p$ is the empirical distribution of the data, $q_\theta$ is the model, and the cross entropy $H(p, q_\theta)$ is proportional to the negative log likelihood. This is also referred to as minimizing the log loss or cross entropy loss.

Reverse KL divergence

When minimizing the reverse KL divergence, we seek the optimal parameters $\theta^*$ as follows:

$$\theta^* = \arg \min_\theta \ D_{KL}(q_\theta \parallel p)$$ $$= \arg \min_\theta \ H(q_\theta, p) - H(q_\theta)$$

Note that the entropy $H(q_\theta)$ depends on $\theta$, so it can't be ignored.

Variational inference--an approximate Bayesian inference strategy--is a notable example of this problem. In this case, $p$ is a (typically intractable) posterior distribution over parameters or latent variables and $q_\theta$ is a tractable approximation.