2
$\begingroup$

I know that mutual information is the Kullback-Leibler divergence between $p(x,y)$ and $p(x)p(y)$. But mutual information is also described as the amount of entropy lost (or, in another sense, the information gained) about $X$ by virtue of knowing about $Y$.

Without just looking at the formula, it's not clear why the additional info about $X$ due to having $Y$ would be the same as the additional information about $Y$ due to having $X$. It's not immediately obvious that predicting $Y$ from the behavior of $X$ is just as easy as predicting $X$ from the behavior of $Y$. While "mutual" is in the name, mutual information is described in terms of learning about $X$ using Y, and so in the same way that e.g. KL divergence (which is described in terms of describing $X$ using $Y$) is asymmetric, the intuition for me would have been that "mutual" information is asymmetric too.

Is there any intuition here without just looking at the formula for $I(X;Y)$?

$\endgroup$

2 Answers 2

2
$\begingroup$

I know that mutual information is the Kullback-Leibler divergence between $p(x,y)$ and $p(x)p(y)$.

Yes and no. Kullback–Leibler divergence between two distributions $p$ and $q$ is defined as

$$ \sum_x p(x) \, \log\Big( \frac{p(x)}{q(x)} \Big) $$

where it is asymmetric because $\tfrac{p(x)}{q(x)} \ne \tfrac{q(x)}{p(x)}$ and because you weight it by $p(x)$ that would give different result as weighting by $q(x)$.

Mutual information between two random variables $X$ and $Y$ is

$$ \sum_{x,y} p_{X,Y}(x, y) \,\log\Big( \frac{p_{X,Y}(x, y)}{p_X(x) \,p_Y(y)} \Big) $$

Notice that $p_{X,Y}(x, y) = p_{Y,X}(y,x)$, same as $p_X(x) \, p_Y(y) = p_Y(y) \, p_X(x)$ so exchanging $X$ with $Y$ would not change the result, it is symmetric.

It would be asymmetric if you changed places of $p_{X,Y}(x, y)$ and $p_X(x) \, p_Y(y)$, but this wouldn't be mutual information anymore, but rather KL divergence. It also wouldn't make much sense, because $p_{X,Y}(x, y)$ is the joint distribution, while using independence in $p_X(x) \, p_Y(y)$ serves as "worst case scenario" that we compare to.

Without just looking at the formula, it's not clear why the additional info about $X$ due to having $Y$ would be the same as the additional information about $Y$ due to having $X$.

This sounds more like the definition of conditional entropy. Mutual information measures “mutual dependence between the two variables” and is symmetric.

$\endgroup$
7
  • $\begingroup$ This isn't an answer: it's a comment that reframes the question. What would your answer be? $\endgroup$
    – whuber
    Feb 11 at 17:35
  • $\begingroup$ Your "No." is wrong. MI is a KL between the joint and product of marginals. $\endgroup$
    – dr.ivanova
    Feb 21 at 10:52
  • $\begingroup$ @dr.ivanova as the answer states, they measure different things, in the sense, they are not the same, and it explains the asymmetry. But ok, edited the wording. $\endgroup$
    – Tim
    Feb 21 at 11:50
  • $\begingroup$ No, they do not measure different things. Mutual information is the KL divergence between the joint and the product of marginals. And vice versa -- you give me the KL between the joint $p_{x,y}$ and $p_x p_y$ and that's exactly the MI(X;Y). Mutual information is symmetric in the densities $p_x$ and $p_y$, not in $X$ and $Y$. I'm not sure what you mean by "exchange $X$ and $Y$", i.e. you cannot swap them to plug in $Y$ in the density $p_x$. $\endgroup$
    – dr.ivanova
    Feb 21 at 15:09
  • $\begingroup$ @dr.ivanova sure you cannot exchange them like you say but you can calculate MI(X, Y) = MI(Y, X) while there is no such symmetry in KL-divergence, because it measures different thing. $\endgroup$
    – Tim
    Feb 21 at 17:30
0
$\begingroup$

You are correct that the $MI$ is the KL-Divergence between the joint $p(x, y)$ and factored marginal distributions $p(x)p(y)$. With the KL-Divergence telling us "how similar" the two distributions are. So, what $MI$ is measuring is the amount of information gain if we update from a model that treats the two variables as independent $p(x)p(y)$ versus one that models the variables joint probability density $p(x, y)$.

If one models the true joint probability density, then we also have access to conditional probability distributions $p(x|y)$ where I like to think of these as "given $Y = y$", how does the distribution of $X$ change? Or, given $X = x$, how does the distribution of $Y$ change? Given we are modeling the joint and factored marginal distributions, we can then re-express $MI$ in terms of joint and conditional entropies:

$I (X; Y ) = H (X) − H (X|Y ) = H (Y ) − H (Y |X)$

Therefore, we can interpret the $MI$ between $X$ and $Y$ as the reduction in uncertainty about $X$ after observing $Y$, or, by symmetry, the reduction in uncertainty about $Y$ after observing $X$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.