8
$\begingroup$

Let $P(x)$ and $Q(x)$ be two pdfs. Let us say that $P(x)$ is the original baseline distribution and $Q(x)$ is the model (or estimate) distribution. I wanted to take the KL Divergence (or the 'distance') between the two distributions.

Which one should I perform: $KL(P,Q)$ or $KL(Q,P)$?

I am using the definition: $KL(p,q) = \displaystyle\int p(x) \ln \dfrac{p(x)}{q(x)}dx$

$\endgroup$
2
  • $\begingroup$ from a purely conceptual perspective, the baseline distribution should be the one with respect to which the expectation is taken (so your first argument). However, note that there are many situations where we do things in the opposite order in practice, such as in Variational Bayes, in order to exploit a relative simplicity of the estimate/model distribution. $\endgroup$ Commented Nov 4, 2023 at 4:24
  • $\begingroup$ The question is pretty thoroughly answered on Tim Vieira's blog. He considers both directions' merits; it agrees with what John Madden said above. $\endgroup$ Commented Nov 4, 2023 at 4:47

2 Answers 2

5
$\begingroup$

If $p(x)$ is the true distribution, and $q(x)$ is your model, then you want to be computing

$$KL(p,q) = \int{p(x)\ln{\frac{p(x)}{q(x)}}}dx.$$

Intuitively, you can see this as the information lost when using $q(x)$ as an approximation of $p(x)$. If you expand the log, it is easier to see that you are subtracting the expected log likelihood of $q(x)$ with respect to $p(x)$, from the true expected log likelihood of $p(x)$ leaving you with a non-negative value expressing said lost information.

When $KL(p,q) > 0$ then the model $q(x)$ is less likely to explain the data compared to the true distribution $p(x)$ (in expectation) and hence a indication of information loss.

When $KL(p,q) = 0$, then $q(x) = p(x)$ for all $x$ satisfying $q(x) > 0$ (converse holds as well). In this scenario, no information is lost.

$\endgroup$
3
$\begingroup$

Assuming that $p_i$ and $q_i$ are discrete, $n$-dimensional probability distributions. In Information Geometry it is proved that for two nearby probability distributions $$q_i=p_i+dp_i$$ with $\sum_i^n dp_i=0$, the infinitesimal distance $ds$ is $$ds \approx \sqrt{2 \; KL(p,q)}.$$ This expression immediately shows that the Kullback-Leibler divergence has the dimension of a length squared, $ds^2$, or an area.

For two probability distributions $p_i$ and $q_i$ which are not close to each other, the proper symmetric(!) geometrical distance is twice the Bhattacharyya angle, or $$s=2\, \arccos \sum_i^n \sqrt{p_i} \; \sqrt{q_i}.$$ This is the arc-length of a geodesic connecting $p_i$ and $q_i$ along a great circle on a $(n-1)$-dimensional hyper-sphere. The distance between two probability distributions is an angle!

For more details see our Kangaroo paper.

$\endgroup$
2
  • $\begingroup$ Oh, wow...that the Kullback-Leibler divergence has the dimension of a length squared (i.e. it's an area) is kind of profound. This is the first time I have come across such an observation - and starts to explain a few things for me. It provides (another) proof that KLD is not a distance. And the "Kangaroo paper" is very interesting. $\endgroup$
    – Mari153
    Commented Jan 7 at 5:52
  • $\begingroup$ One further comment. You mention "two nearby probability distributions" and "two probability distributions...which are not close to each other". How does one decide if two probability distributions are nearby or not close in order to determine which of the respective equations to use? $\endgroup$
    – Mari153
    Commented Jan 7 at 5:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.