I came across a saying that estimates of KL divergence, otherwise known as relative entropy, of the truth of a random variable and its prediction ($y$ and $\hat{y}$) is equal to their cross entropy because entropy + KL divergence = cross entropy or something similar, and that one of the three elements of the formula is fixed or negligible somehow and therefore falls out during estimation or optimization.

And because they are equal, optimizing relative entropy is just the same as using a cross-entropy loss function, which is the more common method. Is this true? or is there a specific application or condition for this to hold?


In many situations, we seek to approximate a 'true' or target distribution $p$ with a model or approximation $q_\theta$ (parameterized by $\theta$). To do this, we search for parameters that minimize a chosen measure of dissimilarity between $p$ and $q_\theta$. The KL divergence is a common choice. It can always be expressed as the difference between a cross entropy term and an entropy term. For two distributions $p_1$ and $p_2$:

$$D_{KL}(p_1 \parallel p_2) = H(p_1, p_2) - H(p_1)$$

The entropy term can always be ignored if we're minimizing the 'forward' KL divergence $D_{KL}(p \parallel q_\theta)$, but cannot be ignored if we're minimizing the 'reverse' KL divergence $D_{KL}(q_\theta \parallel p)$

Forward KL divergence

When minimizing the forward KL divergence, we seek the optimal parameters $\theta^*$ as follows:

$$\theta^* = \arg \min_\theta \ D_{KL}(p \parallel q_\theta)$$ $$= \arg \min_\theta \ H(p, q_\theta) - H(p)$$

Note that the entropy $H(p)$ doesn't depend on $\theta$; it's just an additive constant and can be ignored for the purpose of optimization. That is, the parameters that minimize the forward KL divergence are the same as those that minimize the cross entropy $H(p, q_\theta)$:

$$\theta^* = \arg \min_\theta \ H(p, q_\theta)$$

Maximum likelihood estimation is a particularly common instance of this problem. In this case, $p$ is the empirical distribution of the data, $q_\theta$ is the model, and the cross entropy $H(p, q_\theta)$ is proportional to the negative log likelihood. This is also referred to as minimizing the log loss or cross entropy loss.

Reverse KL divergence

When minimizing the reverse KL divergence, we seek the optimal parameters $\theta^*$ as follows:

$$\theta^* = \arg \min_\theta \ D_{KL}(q_\theta \parallel p)$$ $$= \arg \min_\theta \ H(q_\theta, p) - H(q_\theta)$$

Note that the entropy $H(q_\theta)$ depends on $\theta$, so it can't be ignored.

Variational inference--an approximate Bayesian inference strategy--is a notable example of this problem. In this case, $p$ is a (typically intractable) posterior distribution over parameters or latent variables and $q_\theta$ is a tractable approximation.

  • $\begingroup$ what is the difference between forward and reverse KL divergence? for which data types, conditions and for which applications are each commonly used? not sure what the double vertical bars between $p$ and $q$ signify $\endgroup$
    – develarist
    Aug 4 '20 at 21:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.