4
$\begingroup$

I have trouble understanding KL divergence, where P is probability mass function of true distribution of data and Q is the approximation of P.

The definition of KL divergence is:

$$D_{\mathrm{kl}}(P || Q) = \sum_i \ln\left( \frac{P_i}{Q_i}\right)P_i$$

If I want a symmetrised KL divergence, should it look like the following ?

$$D_{\mathrm{kl}}(P || Q) + D_{\mathrm{kl}}(Q||P) = \sum_i \ln\left( \frac{P_i}{Q_i}\right)P_i + \sum_i \ln\left( \frac{Q_i}{P_i}\right)Q_i$$

Also, which package should I use in R to compute the KL divergence for discrete distributions ? Flexmix or FNN ? Or should I just write my own R function for this?

And what if I don't really know $P$, the true distribution of data?

$\endgroup$
6
$\begingroup$

There are two typical ways to symmetrize the KL divergence:

$$\frac{D(P \| Q) + D(Q \| P)}{2}$$

or

$$\frac{1}{2} D\left(P \; \Big\| \frac{P + Q}{2} \right) + \frac{1}{2} D\left(Q \; \Big\| \frac{P + Q}{2} \right)$$

where $\frac{P+Q}{2}$ means a mixture distribution equally between $P$ and $Q$.

The latter is known as the Jensen-Shannon divergence and has some nice properties. The former is also often nice though, and I use it to good effect in some of my own work.

I don't use R, but it's pretty simple to compute this yourself between two discrete distributions defined by vectors with consistent labeling.

In terms of estimation: the continuous case is kind of hard, but Wang, Kulkarni, and Verdú (2009) give a pretty good estimator based on nearest-neighbor distances. In the discrete case, I'd think that $D(\hat{P} \| \hat{Q})$, where $\hat{P}$, $\hat{Q}$ are any estimate of the density (such as the empirical PMFs) would do fine in most simple cases; I don't know of a direct estimator.

$\endgroup$
  • $\begingroup$ Do I need to do Jacobian transformation in order to obtain the mixture distribution (P + Q) / 2 ??? $\endgroup$ – user1769197 Aug 13 '13 at 16:48
  • 1
    $\begingroup$ If you have them represented as vectors of probabilities, just literally add the two vectors together and divide by two. $\endgroup$ – Dougal Aug 13 '13 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.