16
$\begingroup$

From my readings, I understand that:

  1. Mutual information $\mathit{(MI)}$ is a metric as it meets the triangle inequality, non-negativity, indiscernability and symmetry criteria.
  2. The Kullback–Leibler divergence $\mathit{(D_{KL})}$ is not a metric as it does not obey the triangle inequality

However, one answer on Cross Validated (Information gain, mutual information and related measures) [the second answer], it was shown that mutual information and Kullback–Leibler divergence are equivalent. How can this be given that $\mathit{MI}$ is a metric and $\mathit{D_{KL}}$ is not? I can only assume that I am missing something here.

$\endgroup$
3
  • $\begingroup$ mutual information is not a metric. Variation of information is a metric $\endgroup$
    – develarist
    Commented Sep 11, 2020 at 4:21
  • $\begingroup$ @develarist - thanks for this. This is very interesting as going by a Google search, it is clear that mutual information is treated as a metric by some data analysts. $\endgroup$
    – Mari153
    Commented Sep 11, 2020 at 5:51
  • $\begingroup$ You may be interested in the Christopher Olah - Visual Information Theory post (colah.github.io/posts/2015-09-Visual-Information). He relates mutual information, kl diveregence and other concepts with each other with some visual aids. $\endgroup$
    – Glue
    Commented May 16, 2023 at 15:25

1 Answer 1

18
$\begingroup$

Mutual information is not a metric. A metric $d$ satisfies the identity of indiscernibles: $d(x, y) = 0$ if and only if $x = y$. This is not true of mutual information, which behaves in the opposite manner--zero mutual information implies that two random variables are independent (as far from identical as you can get). And, if two random variables are identical, they have maximal mutual information (as far from zero as you can get).

You're correct that KL divergence is not a metric. It's not symmetric and doesn't satisfy the triangle inequality.

Mutual information and KL divergence are not equivalent. However, the mutual information $I(X, Y)$ between random variables $X$ and $Y$ is given by the KL divergence between the joint distribution $p_{XY}$ and the product of the marginal distributions $p_X \otimes p_Y$ (what the joint distribution would be if $X$ and $Y$ were independent).

$$I(X, Y) = D_{KL}(p_{XY} \parallel p_X \otimes p_Y)$$

Although mutual information is not itself a metric, there are metrics based on it. For example, the variation of information:

$$VI(X, Y) = H(X, Y) - I(X, Y) = H(X) + H(Y) - 2 I(X, Y)$$

where $H(X)$ and $H(Y)$ are the marginal entropies and $H(X, Y)$ is the joint entropy.

$\endgroup$
4
  • 1
    $\begingroup$ This is most interesting as when I type the absolute phrase "mutual information in a metric" into Google, I get something like "about 367,000 results". Some 103 references on Google Scholar also return the use the phrase. So clearly there is some interpretation that MI is being treated as a metric. $\endgroup$
    – Mari153
    Commented Sep 11, 2020 at 5:46
  • 4
    $\begingroup$ @MurrayB By "metric", I mean a formal distance metric, satisfying the conditions you mentioned in the question (symmetry, triangle inequality, identity of indiscernibles). However, it's quite common to use the word "metric" more informally, to describe a way of measuring or quantifying something. I suspect this is probably what's happening in your google results. $\endgroup$
    – user20160
    Commented Sep 11, 2020 at 6:04
  • $\begingroup$ Thanks for this. I was assuming the same. The misuse of terms makes understanding maths, statistics and data analysis all the more challenging. $\endgroup$
    – Mari153
    Commented Sep 11, 2020 at 7:09
  • $\begingroup$ Can you provide the proof for the identity? $\endgroup$
    – Nathan G
    Commented Jun 14, 2023 at 21:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.