# Mutual information between subsets of variables in the multivariate normal distribution

Let $$\vec{X}$$ be a random vector following a multi-variate normal distribution $$P(\vec X)$$ with covariance matrix $$\Sigma$$ and zero means (for simplicity). Consider a partition of $$\vec X$$ into two subsets of variables, $$\vec X = \{\vec X_1, \vec X_2\}$$. What is the mutual information between $$\vec X_1$$ and $$\vec X_2$$:

$$I(\vec X_1; \vec X_2) = \int P(\vec X)\ln \frac{P(\vec X)}{P_1(\vec X_1)P_2(\vec X_2)} \mathrm{d}\vec X$$

where

$$P_1(\vec X_1) = \int P(\vec X)\mathrm{d}\vec X_2, \qquad P_2(\vec X_2) = \int P(\vec X)\mathrm{d}\vec X_1$$

I presume an analytical answer can be given, but I've been having trouble obtaining it.

Actually the answer is trivial to obtain.

$$I(\vec X_1,\vec X_2) = S_1 + S_2 - S_X$$

where $$S_1$$ is the entropy of $$\vec X_1$$, $$S_2$$ the entropy of $$\vec X_2$$, and $$S_X$$ the entropy of the full distribution of $$\vec X$$.

Therefore we obtain the result

$$I(\vec X_1,\vec X_2) = \frac{1}{2} \ln \left( \frac{\det (\Sigma_1) \det (\Sigma_2)}{\det (\Sigma)} \right)$$

where we write the covarinace matrix in the block-form:

$$\Sigma = \left(\begin{array}{cc} \Sigma_{11} & \Sigma_{12}\\ \Sigma_{21} & \Sigma_{22} \end{array}\right)$$

• How can this multivariate form of MI for subsets be extended to its copula entropy equivalent? Dec 10, 2020 at 14:17
• @develarist What do you mean by "its copula entropy equivalent"? Can you expand? Consider asking a new question and notifying me here in the comments. Dec 10, 2020 at 14:22
• Ma (2020) showed that $I(\mathbf{X}) = \iint_{\mathbf{u}} c(\mathbf{u}) \ln c(\mathbf{u}) \, d\mathbf{u}$ where $c(\cdot)$ is a multivariate copula density, and $\mathbf{u}$ are the uniform marginals corresponding to $\mathbf{X}$ arxiv.org/abs/2005.14025 Dec 10, 2020 at 14:33

To see where the results comes from using another approach, first recall that if $$X,Y$$ have joint distribution $$p(X,Y)$$ and marginals $$p(X)$$ and $$p(Y)$$, respectively, then the mutual information is the Kullback-Leibler (KL) divergence between the joint distribution and product of marginal distributions:

$$I(X:Y) = D_{\rm KL}(p(X,Y)\mid\mid p(X)p(Y)).$$

Next, for a pair of $$n$$-dimensional multivariate normal distributions with mean and covariance parameters $$(\mu, \Sigma)$$ and $$(\tilde{\mu}, \tilde{\Sigma})$$, respectively, the KL divergence is given by:

\begin{aligned} &D_{\rm KL}((\mu,\Sigma) \mid\mid (\tilde{\mu}, \tilde{\Sigma})) \\ &= \frac{1}{2} \left[ \log\left( \mathrm{det}(\tilde{\Sigma})/ \mathrm{det}(\Sigma) \right) + \mathrm{tr}\left( \Sigma \tilde{\Sigma}^{-1} \right) - n + (\mu - \tilde{\mu})^\top\tilde{\Sigma}^{-1}(\mu - \tilde{\mu}) \right]. \end{aligned}

Suppose now that the joint distribution $$p(X,Y)$$ is multivariate normal with mean and covariance given by

$$\mu = \begin{bmatrix} \mu_x \\ \mu_y \end{bmatrix}, \qquad \Sigma = \begin{bmatrix} \Sigma_x & \Sigma' \\ \Sigma' & \Sigma_y \end{bmatrix}.$$

The product of marginals $$p(X)p(Y)$$ is also multivariate normal, with mean and covariance given by

$$\tilde{\mu} = \begin{bmatrix} \mu_x \\ \mu_y \end{bmatrix}, \qquad \tilde{\Sigma} = \begin{bmatrix} \Sigma_x & \mathbf{0} \\ \mathbf{0} & \Sigma_y \end{bmatrix}.$$

To complete the computation, we simplify the expression for $$D_{\rm KL}$$ between two multivariate normal distributions whose are parameters are $$(\mu, \Sigma)$$ and $$(\tilde{\mu}, \tilde{\Sigma})$$ as given above, by using the following facts:

1. Since $$\tilde{\Sigma}$$ is a block matrix and $$\Sigma_y$$ is invertible, we have

$$\mathrm{det}(\tilde{\Sigma}) = \mathrm{det}(\Sigma_x - \mathbf{0}\Sigma_y^{-1}\mathbf{0})\mathrm{det}(\Sigma_y) = \mathrm{det}(\Sigma_x)(\Sigma_y).$$

1. From the block structure of $$\tilde{\Sigma}$$, we have

$$\tilde{\Sigma}^{-1} = \begin{bmatrix} \Sigma_x^{-1} & \mathbf{0} \\ \mathbf{0} & \Sigma_y^{-1} \end{bmatrix}$$ which implies that $$\Sigma \tilde{\Sigma}^{-1} = \begin{bmatrix} \Sigma_x & \Sigma' \\ \Sigma' & \Sigma_y \end{bmatrix} \begin{bmatrix} \Sigma_x^{-1} & \mathbf{0} \\ \mathbf{0} & \Sigma_y^{-1} \end{bmatrix} = \begin{bmatrix} \Sigma_x\Sigma_x^{-1} & \Sigma'\Sigma_y \\ \Sigma'\Sigma_x & \Sigma_y^{-1} \end{bmatrix} = \begin{bmatrix} \mathbf{1}_{n_1} & \Sigma'\Sigma_y \\ \Sigma'\Sigma_x & \mathbf{1}_{n_2} \end{bmatrix}$$

where $$n_1$$ and $$n_2$$ are positive integers such that $$n_1 + n_2 = n$$. Thus, it follows that $$\mathrm{tr}\left( \Sigma \tilde{\Sigma}^{-1} \right) - n = n - n = 0.$$

1. Since $$\mu = \tilde{\mu}$$, the right-most term in the expression for $$D_{\rm KL}$$ is equal to zero:

$$\frac{1}{2}(\mu - \tilde{\mu})^\top\tilde{\Sigma}^{-1}(\mu - \tilde{\mu}) = 0.$$

Using these facts, we have shown that

$$I(X:Y) = D_{\rm KL}((\mu,\Sigma) \mid\mid (\tilde{\mu}, \tilde{\Sigma})) = \frac{1}{2} \left[ \log\left( \frac{\mathrm{det}(\Sigma_x)\mathrm{det}(\Sigma_y)}{\mathrm{det}(\Sigma)} \right) \right]$$

as desired.