Symmetric Kullback-Leibler divergence OR Mutual Information as a metric of distance between two distributions?

Question

I need some metric of divergence of two distributions. (They are complex and don't fit with exponential family, normal, log-normal, power-law. Maybe some mixture of that, but I'm not feeling right now figuring that out.)

I'm thinking between Kullback-Leibler divergence and Mutual Information. I don't have any default distribution, therefore I don't like DKL to be asymmetric (Of course I can symmetrize it, but I don't feel confident about it.)

Which one would you choose?

Any other ideas are also welcome!

Answer 1

As far as I can understand you are solving the following problem: there are two analytical distributions $p(x)$ and $q(x)$, and you want to calculate distance between them, $D(p, q)$.

There are a plenty of measures of distance between two distributions:

• Kullback-Liebler
• Mutual information
• Total variation
• Hellinger distance
• Alpha family of divergences and so on

I suggest you to try a few from the list above as all of them are rather easy to implement. In most applications numerical experiments is what give you a key to success. Then you can select one that suits you the best (as I haven't found any requirements for this distance in your question I can't suggest you anything else).

As $p(x)$ and $q(x)$ are complex it is almost impossible that there exists analytical expression for some $D(p, q)$, so you will need a numerical way to calculate those distances. Note, that calculation almost all distances involves numerical integration - so they will be rather imprecise if $x$ dimension is high.

Answer 2

The Kullback-Leibler divergence measures the distance between two distributions: $P(X,Y)$ and $P(X) \cdot P(Y)$. On the other hand, Mutual Information measures the amount of information that one random variable $X$ contains about another random variable $Y$. However, it is possible to relate both metrics as:

$$ I(X,Y) = H(X)+H(Y)-H(X,Y) = D_{KL} \langle P(X,Y) || P(X) \cdot P(Y) \rangle $$

Besides, both metrics are only zero when the r.v. are independent.

$$ P(X,Y) = P(X) \cdot P(Y) \rightarrow \left\{ \begin{matrix} I(X,Y)=H(X)+H(Y)-(H(X)+H(Y))=0 \\ D_{KL} \langle P(X,Y) || P(X) \cdot P(Y) \rangle = 0 \end{matrix}\right. $$

However, as you mentioned, the Kullback-Leibler divergence is not a distance because it is not symmetric and does not follow the triangle inequality. Thus, if you want a symmetric metric and both metrics encode similar information, IMHO the choice of MI seems straightforward.

Answer 3

For convenience, let's say the Kullback-Leibler divergence of $p$ from $q$ is $KDP(p,q)$(not standard notation). All you need to do to symmetrise it is define $f(p,q) = KDP(p,q) + KDP(q,p)$.