# Is $I(X,Y;Z)$ connected with $I(X;Z)$ and $I(Y;Z)$?

Let $I(X,Y;Z)$ be the mutual information between the tuple $X,Y$ and the variable $Z$:

$$I(X,Y;Z) = H(X,Y) - H(X,Y|Z)$$

From the data processing inequality it follows trivially that:

$$I(X;Z) \le I(X,Y;Z),\quad I(X;Y) \le I(X,Y;Z)$$

Is there a more specific connection between $I(X,Y;Z)$ and $I(X;Z)$ and $I(Y;Z)$? For example, if $I(X;Z)=0$, I can see intuitively that $I(X,Y;Z) = I(Y;Z)$, and this is not a consequence of the data processing inequalities; so there must be another connection.

For example, if $I(X;Z)=0$,I can see intuitively that $I(X,Y;Z)=I(Y;Z)$ and this is not a consequence of the data processing inequalities; so there must be another connection.

This intuitive statement is false! The classic example is if Z = X xor Y, and X and Y are independent random binary variables. Each of the mutual information is zero, $I(X;Z)=I(Y;Z)=0$, but $(X,Y;Z)=1$. This is the classic case of "synergistic" variables. I.e., X and Y together have information that neither has individually. A different (and more useful, I think) breakdown of these types of relationships is given by Beer & Williams in their classic paper introducing "Partial Information Decomposition". Though, please note, that the measure of redundancy that they propose their is not unique, some followup papers from Griffith & Koch, and others, consider this.

• Thanks. It also follows from the equations in my answer that my original example is wrong :) – becko Feb 25 '16 at 14:17
• Right! I didn't catch that. The condition for additivity, $I(X,Y;Z) = I(X;Z) + I(Y;Z)$, requires that something called the interaction information: en.wikipedia.org/wiki/Interaction_information is zero. – Greg Ver Steeg Feb 26 '16 at 15:22

## $X$ and $Y$ independent

The simplest case is when $X$ and $Y$ are independent. In this case $H(X,Y) = H(X) + H(Y)$ and $H(X,Y|Z) = H(X|Z) + H(Y|Z)$ (note that I am assuming that $X,Y$ are independent even when $Z$ is given). Then it follows from the identities:

$$I(X,Y;Z) = H(X,Y) - H(X,Y|Z)$$ $$I(X;Z) = H(X) - H(X|Z)$$ $$I(Y;Z) = H(Y) - H(Y|Z)$$

that

$$I(X,Y;Z) = I(X;Z) + I(Y;Z)$$

## $X$ and $Y$ not independent

In general, $X$ and $Y$ may not be independent, and we have:

$$H(X,Y)=H(X)+H(Y)-I(X;Y)$$ $$H(X,Y|Z)=H(X|Z)+H(Y|Z)-I(X;Y|Z)$$

so it follows that:

$$I(X,Y;Z) = I(X;Z) + I(Y;Z) + I(X;Y;Z)$$

where $I(X;Y;Z) = I(X;Y|Z) - I(X;Y)$ is the interaction information, and measures the information shared by all 3 variables. It can be positive, zero, or negative, so not much else can be said. Intuitively, $I(X;Z) + I(Y;Z)$ only measures the information that $X$ and $Y$ each give by itself, but misses the additional information gained by considering $X$ and $Y$ together (if $I(X;Y;Z)$ is positive), or over counts the common information that both $X$ and $Y$ give about $Z$ (if $I(X;Y;Z)$ is negative); in reality, $I(X;Y;Z)$ measures the balance between these two effects.

Another way to write this result is using the chain rule for conditional entropy:

$$I(X,Y;Z) = I(X;Z) + I(Y;Z|X)$$

However in this last form the explicit connection between $I(X,Y;Z)$ and $I(X;Z),I(Y;Z)$ is lost.