Let $I(X,Y;Z)$ be the mutual information between the tuple $X,Y$ and the variable $Z$:

$$I(X,Y;Z) = H(X,Y) - H(X,Y|Z)$$

From the data processing inequality it follows trivially that:

$$I(X;Z) \le I(X,Y;Z),\quad I(X;Y) \le I(X,Y;Z)$$

Is there a more specific connection between $I(X,Y;Z)$ and $I(X;Z)$ and $I(Y;Z)$? For example, if $I(X;Z)=0$, I can see intuitively that $I(X,Y;Z) = I(Y;Z)$, and this is not a consequence of the data processing inequalities; so there must be another connection.


I just want to add one thing to becko's nice answer.

For example, if $I(X;Z)=0$,I can see intuitively that $I(X,Y;Z)=I(Y;Z)$ and this is not a consequence of the data processing inequalities; so there must be another connection.

This intuitive statement is false! The classic example is if Z = X xor Y, and X and Y are independent random binary variables. Each of the mutual information is zero, $I(X;Z)=I(Y;Z)=0$, but $(X,Y;Z)=1$. This is the classic case of "synergistic" variables. I.e., X and Y together have information that neither has individually. A different (and more useful, I think) breakdown of these types of relationships is given by Beer & Williams in their classic paper introducing "Partial Information Decomposition". Though, please note, that the measure of redundancy that they propose their is not unique, some followup papers from Griffith & Koch, and others, consider this.

  • $\begingroup$ Thanks. It also follows from the equations in my answer that my original example is wrong :) $\endgroup$ – becko Feb 25 '16 at 14:17
  • $\begingroup$ Right! I didn't catch that. The condition for additivity, $I(X,Y;Z) = I(X;Z) + I(Y;Z)$, requires that something called the interaction information: en.wikipedia.org/wiki/Interaction_information is zero. $\endgroup$ – Greg Ver Steeg Feb 26 '16 at 15:22

$X$ and $Y$ independent

The simplest case is when $X$ and $Y$ are independent. In this case $H(X,Y) = H(X) + H(Y)$ and $H(X,Y|Z) = H(X|Z) + H(Y|Z)$ (note that I am assuming that $X,Y$ are independent even when $Z$ is given). Then it follows from the identities:

$$I(X,Y;Z) = H(X,Y) - H(X,Y|Z)$$ $$I(X;Z) = H(X) - H(X|Z)$$ $$I(Y;Z) = H(Y) - H(Y|Z)$$


$$I(X,Y;Z) = I(X;Z) + I(Y;Z)$$

$X$ and $Y$ not independent

In general, $X$ and $Y$ may not be independent, and we have:

$$H(X,Y)=H(X)+H(Y)-I(X;Y)$$ $$H(X,Y|Z)=H(X|Z)+H(Y|Z)-I(X;Y|Z)$$

so it follows that:

$$I(X,Y;Z) = I(X;Z) + I(Y;Z) + I(X;Y;Z)$$

where $I(X;Y;Z) = I(X;Y|Z) - I(X;Y)$ is the interaction information, and measures the information shared by all 3 variables. It can be positive, zero, or negative, so not much else can be said. Intuitively, $I(X;Z) + I(Y;Z)$ only measures the information that $X$ and $Y$ each give by itself, but misses the additional information gained by considering $X$ and $Y$ together (if $I(X;Y;Z)$ is positive), or over counts the common information that both $X$ and $Y$ give about $Z$ (if $I(X;Y;Z)$ is negative); in reality, $I(X;Y;Z)$ measures the balance between these two effects.

Another way to write this result is using the chain rule for conditional entropy:

$$I(X,Y;Z) = I(X;Z) + I(Y;Z|X)$$

However in this last form the explicit connection between $I(X,Y;Z)$ and $I(X;Z),I(Y;Z)$ is lost.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.