On the Wikipedia of mutual information it says that $I(X;Y)=0$ if and only if $X$ and $Y$ are independent.

I can proof that if $X$ and $Y$ are independent, then $I(X;Y)=0$, because $p(x,y)=p(x)p(y)$. But how can one prove that if $I(X;Y)=0$, then $X$ and $Y$ are independent?


Here's my take. In the discrete case,

$$ \operatorname{I}(X;Y) = \sum_{y \in \mathcal Y} \sum_{x \in \mathcal X} {p_{(X,Y)}(x,y) \log{ \left(\frac{p_{(X,Y)}(x,y)}{p_X(x)\,p_Y(y)} \right) } } $$

So $I(X;Y)=0$ when, at all points, either:

  1. ${p_{(X,Y)}(x,y)} = 0$, or
  2. $\log{ \left(\frac{p_{(X,Y)}(x,y)}{p_X(x)\,p_Y(y)} \right) } \;=0$

It's worth noting that MI is always negative or 0, and we can't get a positive logarithm at any point, because the joint probability is always a subset of the marginals, so we don't need to worry about sums cancelling each other out; just that each $\operatorname{I}(X;Y)=0\, \forall X,Y$. (And of course, by definition, any data point can't reduce the total amount of information.)

The first case says the joint probability is 0 in those cases. It's essentially saying that the two events can't happen together, so I think we're ok just treating these as impossible or undefined events.

The second case requires that $\frac{p_{(X,Y)}\,(x,y)}{p_X(x)\,p_Y(y)} = 1$, which implies $p_{(X,Y)} = p_X(x)\,p_Y(y)$, which is independence, for all cases where both events can happen together.

The logic is the same in the continuous case.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.