# KL Divergence, which pdf is which?

Let $$P(x)$$ and $$Q(x)$$ be two pdfs. Let us say that $$P(x)$$ is the original baseline distribution and $$Q(x)$$ is the model (or estimate) distribution. I wanted to take the KL Divergence (or the 'distance') between the two distributions.

Which one should I perform: $$KL(P,Q)$$ or $$KL(Q,P)$$?

I am using the definition: $$KL(p,q) = \displaystyle\int p(x) \ln \dfrac{p(x)}{q(x)}dx$$

• from a purely conceptual perspective, the baseline distribution should be the one with respect to which the expectation is taken (so your first argument). However, note that there are many situations where we do things in the opposite order in practice, such as in Variational Bayes, in order to exploit a relative simplicity of the estimate/model distribution. Nov 4 at 4:24
• The question is pretty thoroughly answered on Tim Vieira's blog. He considers both directions' merits; it agrees with what John Madden said above. Nov 4 at 4:47

If $$p(x)$$ is the true distribution, and $$q(x)$$ is your model, then you want to be computing

$$KL(p,q) = \int{p(x)\ln{\frac{p(x)}{q(x)}}}dx.$$

Intuitively, you can see this as the information lost when using $$q(x)$$ as an approximation of $$p(x)$$. If you expand the log, it is easier to see that you are subtracting the expected log likelihood of $$q(x)$$ with respect to $$p(x)$$, from the true expected log likelihood of $$p(x)$$ leaving you with a non-negative value expressing said lost information.

When $$KL(p,q) > 0$$ then the model $$q(x)$$ is less likely to explain the data compared to the true distribution $$p(x)$$ (in expectation) and hence a indication of information loss.

When $$KL(p,q) = 0$$, then $$q(x) = p(x)$$ for all $$x$$ satisfying $$q(x) > 0$$ (converse holds as well). In this scenario, no information is lost.

Assuming that $$p_i$$ and $$q_i$$ are discrete, $$n$$-dimensional probability distributions. In Information Geometry it is proved that for two nearby probability distributions $$q_i=p_i+dp_i$$ with $$\sum_i^n dp_i=0$$, the infinitesimal distance $$ds$$ is $$ds \approx \sqrt{2 \; KL(p,q)}.$$ This expression immediately shows that the Kullback-Leibler divergence has the dimension of a length squared, $$ds^2$$, or an area.

For two probability distributions $$p_i$$ and $$q_i$$ which are not close to each other, the proper symmetric(!) geometrical distance is twice the Bhattacharyya angle, or $$s=2\, \arccos \sum_i^n \sqrt{p_i} \; \sqrt{q_i}.$$ This is the arc-length of a geodesic connecting $$p_i$$ and $$q_i$$ along a great circle on a $$(n-1)$$-dimensional hyper-sphere. The distance between two probability distributions is an angle!

For more details see our Kangaroo paper.