# Rewriting the mutual information of a linear model through conditional expectation

I am reading the following paper:

http://web.mit.edu/18.325/www/telatar_capacity.pdf

In this paper we have the following linear model, with $$\mathbf{n}$$ being additive noise:

$$\begin{equation} \mathbf{y = Hx + n} \quad \mathbf{H}\in\mathbb{C}^{r\times t}, \mathbf{x}\in\mathbb{R}^t,\mathbf{y}\in\mathbb{C}^r \end{equation}$$

The aim on page 10 (section 4.1) is to calculate the mutual information $$\mathcal{I}(x;(y,H))=\mathcal{I}(\text{input;output})$$. Now the paper performs the following steps:

\begin{align} \mathcal{I}(\mathbf{x};\mathbf{(y,H)}) &= \mathcal{I}(\mathbf{x};\mathbf{H}) +\mathcal{I}(\mathbf{x};\mathbf{y}|\mathbf{H})\\ &=\mathcal{I}(\mathbf{x};\mathbf{y}|\mathbf{H})\\ &= \mathbb{E}[\mathcal{I}(\mathbf{x};\mathbf{y}|\mathbf{H}=H)] \end{align}

Now I have difficulties understanding the transiation that is $$\mathcal{I}(\mathbf{x};\mathbf{y}|\mathbf{H}) = \mathbb{E}[\mathcal{I}(\mathbf{x};\mathbf{y}|\mathbf{H}=H)]$$.

1. Could somebody explain/clarify this point? I thought conditioning on RVs $$\mathbf{y}|\mathbf{H}$$ implied I was doing the following: $$\mathbf{y}|\mathbf{H}=H$$, but in the above statement these are separate ideas.

2. Also another question is: what probability distribution is the expectation being taken over?

3. Lastly, would the same equations be obtainable if we consider $$\mathcal{I}(\mathbf{x};\mathbf{y})$$ as this is more direct for the input-output relationship rather than with the $$\mathbf{H}$$ floating around there in addition. i.e. consider: $$\mathcal{I}(\mathbf{x};\mathbf{y})$$ instead of $$\mathcal{I}(\mathbf{x};(\mathbf{y},\mathbf{H}))$$?

Although I find the notation used in the equation quite awkward, I will show you how to establish that result using the same notation. The mutual information is the expected value of the logarithm of the ratio of the joint density divided by the product of the marginal densities. Thus, for the variables in question, you have:

$$\mathcal{I}(\mathbf{x};\mathbf{(y,H)}) =\mathbb{E} \Bigg( \log \bigg(\frac{p(\mathbf{x}, \mathbf{y}, \mathbf{H})}{p(\mathbf{x}) p(\mathbf{y}, \mathbf{H})} \bigg) \Bigg).$$

Now, note that:

$$\frac{p(\mathbf{x}, \mathbf{y}, \mathbf{H})}{p(\mathbf{x}) p(\mathbf{y}, \mathbf{H})} = \frac{p(\mathbf{x}, \mathbf{y}, \mathbf{H})}{p(\mathbf{x}) p(\mathbf{y}, \mathbf{H})} \cdot \frac{p(\mathbf{H})}{p(\mathbf{H})} = \frac{p(\mathbf{x}, \mathbf{y}| \mathbf{H})}{p(\mathbf{x}) p(\mathbf{y}| \mathbf{H})}.$$

Thus, using the law of iterated expectation we can write the mutual information as:

\begin{aligned} \mathcal{I}(\mathbf{x};\mathbf{(y,H)}) &= \mathbb{E} \Bigg( \log \bigg(\frac{p(\mathbf{x}, \mathbf{y}, \mathbf{H})}{p(\mathbf{x}) p(\mathbf{y}, \mathbf{H})} \bigg) \Bigg) \\[6pt] &= \mathbb{E} \Bigg( \log \bigg(\frac{p(\mathbf{x}, \mathbf{y}| \mathbf{H})}{p(\mathbf{x}) p(\mathbf{y}| \mathbf{H})} \bigg) \Bigg) \\[6pt] &= \mathbb{E}_H \Bigg( \mathbb{E} \Bigg( \log \bigg(\frac{p(\mathbf{x}, \mathbf{y}| \mathbf{H}=H)}{p(\mathbf{x}) p(\mathbf{y}| \mathbf{H}=H)} \bigg) \Bigg) \Bigg) \\[6pt] &= \mathbb{E}_H \Big( \mathcal{I}(\mathbf{x};\mathbf{y}|\mathbf{H} = H) \Big). \\[6pt] \end{aligned}

As you can see, nothing in this working requires you to be using a linear model. It is a general result that applies regardless of the relationship of the random variables under consideration.

1.

To understand the answer to your first question, we first need to understand what mutual information is and how it is different from conditional mutual information.

For random variables $$X$$ and $$Y$$ with distributions $$P_{X}$$ and $$P_{Y}$$ respectively, mutual information $$I(X; Y)$$ is a measure of independence between $$P_{X}$$ and $$P_{Y}$$, defined as the KL-divergence between the joint distribution $$P_{(X, Y)}$$ and the product of marginals $$P_{X} \otimes P_{Y}$$ as follows: $$I(X; Y) = D_{KL}(P_{(X, Y)} || P_{X} \otimes P_{Y}).$$ If $$P_{(X, Y)} = P_{X} \otimes P_{Y}$$ then then $$X$$ and $$Y$$ are independent and hence $$I(X; Y) = 0$$ and the same goes from the other direction. In other words, $$I(X; Y)$$ measures the amount of information about $$X$$ we get after observing $$Y$$ and vice-versa.

Now let $$Z$$ be some other random variable with distribution $$P_{Z}$$. Suppose we have observed $$Z = z$$. Conditionally on this new information, the mutual information between $$X$$ and $$Y$$ may have changed. For example, if $$Z = (X, Y)$$, then observing $$X$$ does not provide any new information about $$Y$$, since all of it is already contained in the random variable $$Z$$. To account for this new information, we can compute a refined measure of independence between the distributions of $$X$$ and $$Y$$. In particular, we can let $$P_{X \mid Z=z}$$ and $$P_{Y \mid Z = z}$$ denote the distribution of $$X$$ and $$Y$$ conditionally on $$Z = z$$ and let $$I(X;Y \mid Z = z) = D_{KL}(P_{(X, Y) \mid Z = z} || P_{X \mid Z=z} \otimes P_{Y \mid Z=z}).$$

With the above notation at hand, we can now say what the difference between $$I(X;Y \mid Z)$$ and $$I(X;Y \mid Z = z)$$ is. The former is formally a random-variable, while the latter is a number. The conditional mutual information is defined to be the expected value of $$I(X;Y \mid Z)$$ -- that is, what is the average measure of independence between $$X$$ and $$Y$$ after observing $$Z$$, which is typically denoted by the following rather unfortunate, overloaded notation: $$I(X;Y\mid Z) = \mathbb{E}_{z \sim P_{Z}} I(X;Y \mid Z = z).$$

2.

Using the above notation $$Z = \mathbf{H}$$, hence the expectation is taken with respect to $$\mathbf{H} \sim P_{\mathbf{H}}$$. That is, the design matrix $$\mathbf{H}$$ is assumed to be random and follows distribution $$P_{\mathbf{H}}$$.

3.

$$I(\mathbf{x}; \mathbf{y})$$ and $$I(\mathbf{x}; (\mathbf{y}, \mathbf{H}))$$ have very different meanings! To appreciate the difference, consider a setting where $$r = t$$ and suppose that $$\mathbf{H}$$ is always invertible (that is, $$P_{\mathbf{H}}$$ is supported on a set of invertible square $$t \times t$$ matrices). Further, let the additive noise term $$\mathbf{n}$$ be always $$0$$. Then \begin{align} \tag{*} \mathbf{y} &= \mathbf{Hx},\\ \mathbf{x} &= \mathbf{H^{-1}y}. \end{align} Hence, using the formulas you wrote down we have $$I(\mathbf{x}, (\mathbf{y}, \mathbf{H})) = I(\mathbf{x}, \mathbf{y} \mid \mathbf{H})$$ which means that $$I(\mathbf{x}, (\mathbf{y}, \mathbf{H}))$$ is large, because knowing $$\mathbf{H}$$ and one of $$\mathbf{x}, \mathbf{y}$$ the other one can be computed exactly using Equation (*) above.

On the other hand, $$I(\mathbf{x};\mathbf{y})$$ is going to be small, because without the knowledge of $$\mathbf{H}$$, $$\mathbf{x}$$ says nothing about $$\mathbf{y}$$ and vice-versa.