I need some metric of divergence of two distributions. (They are complex and don't fit with exponential family, normal, log-normal, power-law. Maybe some mixture of that, but I'm not feeling right now figuring that out.)

I'm thinking between Kullback-Leibler divergence and Mutual Information. I don't have any default distribution, therefore I don't like DKL to be asymmetric (Of course I can symmetrize it, but I don't feel confident about it.)

Which one would you choose?

Any other ideas are also welcome!

  • 1
    $\begingroup$ $KL(\mu,\nu)+KL(\nu,\mu)$ or $\min\{KL(\mu,\nu),KL(\nu,\mu)\}$ are two possible symmetrizations of the Kullback-Leibler divergence. $\endgroup$ Sep 9 '15 at 15:36

As far as I can understand you are solving the following problem: there are two analytical distributions $p(x)$ and $q(x)$, and you want to calculate distance between them, $D(p, q)$.

There are a plenty of measures of distance between two distributions:

I suggest you to try a few from the list above as all of them are rather easy to implement. In most applications numerical experiments is what give you a key to success. Then you can select one that suits you the best (as I haven't found any requirements for this distance in your question I can't suggest you anything else).

As $p(x)$ and $q(x)$ are complex it is almost impossible that there exists analytical expression for some $D(p, q)$, so you will need a numerical way to calculate those distances. Note, that calculation almost all distances involves numerical integration - so they will be rather imprecise if $x$ dimension is high.


The Kullback-Leibler divergence measures the distance between two distributions: $P(X,Y)$ and $P(X) \cdot P(Y)$. On the other hand, Mutual Information measures the amount of information that one random variable $X$ contains about another random variable $Y$. However, it is possible to relate both metrics as:

$$ I(X,Y) = H(X)+H(Y)-H(X,Y) = D_{KL} \langle P(X,Y) || P(X) \cdot P(Y) \rangle $$

Besides, both metrics are only zero when the r.v. are independent.

$$ P(X,Y) = P(X) \cdot P(Y) \rightarrow \left\{ \begin{matrix} I(X,Y)=H(X)+H(Y)-(H(X)+H(Y))=0 \\ D_{KL} \langle P(X,Y) || P(X) \cdot P(Y) \rangle = 0 \end{matrix}\right. $$

However, as you mentioned, the Kullback-Leibler divergence is not a distance because it is not symmetric and does not follow the triangle inequality. Thus, if you want a symmetric metric and both metrics encode similar information, IMHO the choice of MI seems straightforward.


For convenience, let's say the Kullback-Leibler divergence of $p$ from $q$ is $KDP(p,q)$(not standard notation). All you need to do to symmetrise it is define $f(p,q) = KDP(p,q) + KDP(q,p)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.