Minimising the KL divergence between your model distribution and the true data distribution is equivalent to maximising the (log-) likelihood.

In machine learning, we often want to create a model with some parameter(s) $\theta$ that maximises the likelihood of some distribution. I have a couple of questions regarding how minimising other divergence measures optimise our model. In particular:

  1. Does the Jensen Shannon Divergence also maximise the likelihood? If not what does it maximise?
  2. Does the reverse KL divergence also maximise the likelihood? If not what does it maximise?


As you can see from the figure below from this paper , the KL and JSD have different optimal solutions, so if minimising the KL is equivalent to optimising the likelihood, then the same cannot necessarily be the case for JSD.

enter image description here


First, it is important to clarify a few things.

  1. The KL divergence is a dissimilarity between two distributions, so it cannot maximize the likelihood, which is a function of a single distribution.
  2. Given a reference distribution $P(\cdot)$, the value of $\theta$ that minimizes $\text{KL}(P(\cdot)||Q(\cdot|\theta))$ is not the one that maximizes the likelihood. Actually, there is no likelihood because there is no observed value.

So, saying that minimizing the KL divergence is equivalent to maximizing the log-likelihood can only mean that choosing $\hat{\theta}$ so as to maximize $Q(x_1, \ldots, x_n|\theta)$, ensures that $ \hat{\theta} \rightarrow \theta^*$, where

$$\theta^* = \text{argmin}_\theta \text{ KL}(P(\cdot)||Q(\cdot|\theta)).$$

This is true under some usual regularity conditions. To see this, assume that we compute $Q(x_1, \ldots, x_n|\theta)$, but the sample $x_1, \ldots, x_n$ is actually drawn from $P(\cdot)$. The expected value of the log-likelihood is then

$$\int P(x_1, \ldots, x_n) \log Q(x_1, \ldots, x_n|\theta) dx_1 \ldots dx_n.$$

Maximizing this value with respect to $\theta$ is he same as minimizing

$$\text{KL}(P(\cdot)||Q(\cdot|\theta)) = \int P(x_1, \ldots, x_n) \log \frac{P(x_1, \ldots, x_n)}{Q(x_1, \ldots, x_n|\theta)}dx_1 \ldots dx_n.$$

This is not an actual proof, but this gives you the main idea. Now, there is no reason why $\theta^*$ should also minimize

$$\text{KL}(Q(\cdot|\theta)||P(\cdot)) = \int Q(x_1, \ldots, x_n|\theta) \log \frac{Q(x_1, \ldots, x_n|\theta)}{P(x_1, \ldots, x_n)}dx_1 \ldots dx_n.$$

Your question actually provides a counter-example of this, so it is clear that the value of $\theta$ that minimizes the reverse KL divergence is in general not the same as the maximum likelihood estimate (and thus the same goes for the Jensen-Shannon divergence).

What those values minimize is not so well defined. From the argument above, you can see that the minimum of the reverse KL divergence corresponds to computing the likelihood as $P(x_1, \ldots, x_n)$ when $x_1, \ldots, x_n$ is actually drawn from $Q(\cdot|\theta)$, while trying to keep the entropy of $Q(\cdot|\theta)$ as high as possible. The interpretation is not straightforward, but we can think of it as trying to find a "simple" distribution $Q(\cdot|\theta)$ that would "explain" the observations $x_1, \ldots, x_n$ coming from a more complex distribution $P(\cdot)$. This is a typical task of variational inference.

The Jensen-Shannon divergence is the average of the two, so one can think of finding a minimum as "a little bit of both", meaning something in between the maximum likelihood estimate and a "simple explanation" for the data.

  • 1
    $\begingroup$ Thanks for your highly descriptive and informative answer. I am a little confused though by your last two sentences. If you look at the very first figure on arxiv.org/abs/1511.01844 you can see that the KLD and the JSD converge to different solutions so their optimal objectives can't be the same. I.e. they can't both be equivalent. $\endgroup$
    – Mellow
    Apr 27 '19 at 11:19
  • 1
    $\begingroup$ Hi @gui11aume, I have updated my original post to add the figure. $\endgroup$
    – Mellow
    Apr 27 '19 at 11:54
  • 1
    $\begingroup$ OK, I see your point. My answer is incomplete because it assumes that we have the same family of distributions in the KL divergence (in line with the blog post you linked to). I will update the answer. $\endgroup$
    – gui11aume
    Apr 27 '19 at 12:01
  • 1
    $\begingroup$ Thanks for your updated awesome answer. I think there's a small typo - I think the reverse KL tries to increase the entropy of Q, as $KL(Q||P)$ = $E_Q[log Q] - E_Q[log P]$ = $-H[Q] - E_Q[log P]$. So minimising the KL would maximise the entropy of Q $\endgroup$
    – Mellow
    Apr 28 '19 at 9:35
  • 1
    $\begingroup$ Yes of course. I wrote too fast. Simple distribution have high entropy, not low entropy. Thanks for spotting! $\endgroup$
    – gui11aume
    Apr 28 '19 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.