Consider this simple boolean relationship between the binary variables A, B and C:


I.e. $A$ is 1 if either of $B$ or $C$ are 1, otherwise $A$ is always 0.

We also have these extra conditions:



where $PMI(B;C)$ denotes the pointwise mutual information between the "1" outcomes of $B$ and $C$.

Now, unless I'm mistaken (I have limited training in statistics), it follows that:


That is, the amount of pointwise mutual information between the "1" outcomes of $A$ and $B$ is the same as the amount of pointwise mutual info between the "1" outcomes of $A$ and $C$, and is equal to the information-content of the "1" outcome of $A$, i.e. $-log(P(A))$.

But how is it possible for both $B$ and $C$ to both (individually) "completely overlap" with $A$ in terms of their pointwise mutual information, and yet for $B$ and $C$ to not "overlap" with one another at all? When I imagine the meaning of $A=B∥C$ in terms of the pointwise mutual information, this is what seems to make sense:

A = B or C

That is, all of $A$'s pointwise information is "contained within" both $B$ and $C$, and thus $B$ and $C$ must "overlap" via $A$'s pointwise information.

But that obviously doesn't make sense because we've explicitely specified that $B$ and $C$ are independent: $PMI(B;C)=0$ (and this is not made impossible by the relationship $A=B∥C$).

I previously thought that we could say that the "1" outcome of $A$ "contains" some bits (using base 2), and it can "share" those bits with outcomes of other variables (measured via pointwise mutual information), but this example seems to suggest that this simple understanding is wrong/incomplete.

So my question is: Where is the flaw in my understanding of information which causes my intuition to differ from what's obviously true in this simple example?


1 Answer 1


I think to understand the differences it is first good to get a good notion of what is entropy and conditional entropy. There are many ways to explain it, but the intuition of entropy is that it is related to the surprise. A deterministic outcome has no surprise, while a fair coin flip has exactly 1 bits of surprise.

Here $ B $ and $ C $ are essentially coin flips, if they are unbiased $ H(A) = H(B) = 1 $, but the value does not matter. They are independent coin flips, so flipping A won't tell us the outcome of B, and the converse. That is represented by $ H(B|C) = H(B) $ and $ H(C|B) = H(C) $ which is the conditional entropy.

Now I advise to regrasp the intuition of mutual infomration using this interpreation rather than the one that you have drawn. Then some flaws of this reasoning should become apparent,

  1. You cannot state something like $ I(A) $, because mutual information is interpreted with two random variables.
  2. $ I(A;B) = I(A;C) $ should hold true because they are similiar random variables. This means they hold true not because $ A = B $ but because $ H(A) = H(B) $

So in this case what just happens is that the the two random variables $ B $ and $ C $ contain complementary information, and thus the mutual information is 0. That is in fact implied by the notion of independence. Thus the diagrams are not right the way you have drawn it. You can only draw such diagrams for entropic amounts, and then informally $ H(A|B) \subset H(B) $ and $ H(A|C) \subset H(C) $, and the two sets are disjoint.

  • $\begingroup$ Ah I think I'm confusing "point-wise mutual information" with mutual information. Each time I wrote $I(...)$, I meant $PMI(...)$ and was referring to the "1" outcome of A, B and C. Does my question make more sense now (especially what I meant by $I(A)$)? In case you do not have time to follow up, thank you for your help! You've definitely set me on the path to fixing my intuitions here. $\endgroup$
    – user136692
    May 6, 2019 at 8:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.