3
$\begingroup$

I want to compute the mutual information between two discrete random variables $x_1,x_2$ whose PMFs are affected by measurements resp. $z_1,z_2$ (one measurement per variable) in a known way. Based on the definition we have

$ \displaystyle{I(x_1;x_2\vert z_1,z_2)=\sum_{x_1,x_2,z_1,z_2}p_{X_1,X_2, Z_1,Z_2}(x_1,x_2,z_1,z_2) \log\frac{p_{Z_1,Z_2}(z_1,z_2)p_{X_1,X_2,Z_1,Z_2}(x_1,x_2,z_1,z_2)}{p_{X_1,Z_1,Z_2}(x_1,z_1,z_2) p_{X_2,Z_1,Z_2}(x_2,z_1,z_2)}} $

I am trying to make sense of this definition: it's not clear to me why the variable that we condition on is "integrated away" by $\sum_{z_1,z_2}$. In other words, it seems that in order to use this definition I need some prior for the measurements $z_1$ and $z_2$. Maybe I should just use the different quantity

$ \displaystyle{I'(x_1;x_2\vert z_1,z_2)=\sum_{x_1,x_2} p_{X_1,X_2,Z_1,Z_2}(x_1,x_2,z_1,z_2) \log\frac{p_{Z_1,Z_2}(z_1,z_2)p_{X_1,X_2,Z_1,Z_2}(x_1,x_2,z_1,z_2)}{p_{X_1,Z_1,Z_2}(x_1,z_1,z_2) p_{X_2,Z_1,Z_2}(x_2,z_1,z_2)}} $

By the way does this quantity have a name?

$\endgroup$
  • 1
    $\begingroup$ There is something weird in your notation. Why do $Y_1$ and $Y_2$ pop up in the summation but do not appear in the input to the MI? $\endgroup$ – Cesare Aug 16 at 10:05
  • $\begingroup$ Can it be that the denominator in your equations should be $p(x_1, z_1, z_2) p(x_2, z_1, z_2)$? $\endgroup$ – Cesare Aug 16 at 10:16
  • $\begingroup$ Thanks for spotting the error @Cesare , I corrected the equations. $\endgroup$ – Arrigo Benedetti Aug 16 at 15:50
1
$\begingroup$

Quantifying dependence, given particular values of $Z_1,Z_2$

First, consider the case where $Z_1,Z_2$ are known to take particular values $z_1,z_2$. Given this knowledge, the conditional dependence between $X_1$ and $X_2$ can be quantified using the KL divergence between the joint distribution and the product of the marginal distributions:

$$D_{KL} \big( p(x_1,x_2 \mid z_1,z_2) \parallel p(x_1 \mid z_1,z_2) p(x_2 \mid z_1,z_2) \big) \tag{1}$$

You can think of this as measuring the dissimilarity between the true joint distribution and what it would be if $X_1$ and $X_2$ were conditionally independent (i.e. the product of the marginals). Notice the similarity between this quantity and the ordinary (unconditional) mutual information. The only difference is that, here, that we're conditioning on $Z_1=z_1, Z_2=z_2$.

Alternatively, using the link between KL divergence and conditional entropy, expression $(1)$ is equivalent to:

$$H(X_1 \mid Z_1=z_1, Z_2=z_2) \ - \ H(X_1 \mid X_2, Z_1=z_1, Z_2=z_2) \tag{2}$$

That is: suppose we already know values $z_1,z_2$. Now, we are additionally told the value of $X_2$. How much does this reduce our uncertainty about $X_1$ on average? (where the average is taken over all possible values of $X_2$).

Conditional mutual information

Expression $(1)$ above is perfectly satisfactory if we're interested in particular values of $Z_1,Z_2$. Alternatively, we might like to quantify the conditional dependence between $X_1,X_2$ over all values of $Z_1,Z_2$. The dependence structure may change depending on these values. This leads to the definition of the conditional mutual information:

$$I(X_1, X_2 \mid Z_1, Z_2) = $$

$$E_{p(z_1, z_2)} \Big[ D_{KL} \big( p(x_1, x_2 \mid z_1, z_2) \parallel p(x_1 \mid z_1, z_2) p(x_2 \mid z_1, z_2) \big) \Big] \tag{2}$$

The conditional mutual information simply averages the KL divergence in expression $(1)$ over all possible values of $Z_1,Z_2$, weighted by the probability of each.

The expression for conditional mutual information given in the question (for discrete variables) is equivalent to expression $(2)$ here. You can verify this by writing out the definition of the KL divergence, then rearranging things using the definition of conditional probability.

The quantity $I'$

Maybe I should just use the different quantity $I'(X_1,X_2 \mid Z_1,Z_2)$ [.....] does this quantity have a name?

I'm not aware of a name for this quantity, but it's proportional to the KL divergence in expression $(1)$ above:

$$p(z_1, z_2) \ I'(X_1,X_2 \mid Z_1,Z_2) =$$

$$D_{KL} \big( p(x_1,x_2 \mid z_1,z_2) \parallel p(x_1 \mid z_1,z_2) p(x_2 \mid z_1,z_2) \big)$$

$p(z_1, z_2)$ is a constant, since $z_1,z_2$ are considered fixed values here. So, as in $(1)$, your quantity $I'$ is measuring the dependence between $X_1,X_2$ given particular values for $Z_1,Z_2$. But, I think $(1)$ is a bit more natural to interpret than $I'$.

$\endgroup$
  • $\begingroup$ It turns out that there is a quantity called pointwise mutual information related to $I'$. So $I'$ is a sort of pointwise conditional mutual information $\endgroup$ – Arrigo Benedetti Aug 19 at 17:57
  • $\begingroup$ @ArrigoBenedetti I see the analogy you're trying to make, but I don't think it's quite right to call $I'$ a type of pointwise MI. The difference is that pointwise MI is defined for specific values of the variables whose dependence we want to quantify ($X_1,X_2$ in this case). Whereas here, you want to average over all values of $X_1,X_2$ (as in regular MI) but consider fixed values for the variables upon which the dependence is conditioned ($Z_1,Z_2$). $\endgroup$ – user20160 Aug 19 at 19:55
2
$\begingroup$

You are trying to compute a conditional mutual information. If you call $X$ your $X_1$, $Y$ your $X_2$ and $\mathbf{Z} = [Z_1, Z_2]$, you obtain the same notation as in the wiki. In your first equation you are computing $I(X;Y|\mathbf{Z})$ (note that upper-case letters are usually used to denote the random variable while lower-case denote specific outcomes). In your second equation you are computing only one of the terms that make up the conditional mutual information, those corresponding to a chose outcome $\mathbf{z} = [z_1, z_2]$. This is usually written using the notation $I(X;Y|\mathbf{z})$.

$\endgroup$
  • $\begingroup$ I think $I(X; Y \mid z)$ would be more commonly defined such that the conditional mutual information is its expectation (w.r.t. $Z$). The OP's second equation doesn't quite match this, since the conditional mutual information is obtained by summing it over $Z$, rather than taking the expected value. $\endgroup$ – user20160 Aug 17 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.