Let $\vec{X}$ be a random vector following a multi-variate normal distribution $P(\vec X)$ with covariance matrix $\Sigma$ and zero means (for simplicity). Consider a partition of $\vec X$ into two subsets of variables, $\vec X = \{\vec X_1, \vec X_2\}$. What is the mutual information between $\vec X_1$ and $\vec X_2$:

$$I(\vec X_1; \vec X_2) = \int P(\vec X)\ln \frac{P(\vec X)}{P_1(\vec X_1)P_2(\vec X_2)} \mathrm{d}\vec X$$


$$P_1(\vec X_1) = \int P(\vec X)\mathrm{d}\vec X_2, \qquad P_2(\vec X_2) = \int P(\vec X)\mathrm{d}\vec X_1 $$

I presume an analytical answer can be given, but I've been having trouble obtaining it.


Actually the answer is trivial to obtain.

$$I(\vec X_1,\vec X_2) = S_1 + S_2 - S_X$$

where $S_1$ is the entropy of $\vec X_1$, $S_2$ the entropy of $\vec X_2$, and $S_X$ the entropy of the full distribution of $\vec X$.

Therefore we obtain the result

$$I(\vec X_1,\vec X_2) = \frac{1}{2} \ln \left( \frac{\det (\Sigma_1) \det (\Sigma_2)}{\det (\Sigma)} \right)$$

where we write the covarinace matrix in the block-form:

$$\Sigma = \left(\begin{array}{cc} \Sigma_{11} & \Sigma_{12}\\ \Sigma_{21} & \Sigma_{22} \end{array}\right)$$

  • $\begingroup$ How can this multivariate form of MI for subsets be extended to its copula entropy equivalent? $\endgroup$ – develarist Dec 10 '20 at 14:17
  • $\begingroup$ @develarist What do you mean by "its copula entropy equivalent"? Can you expand? Consider asking a new question and notifying me here in the comments. $\endgroup$ – becko Dec 10 '20 at 14:22
  • $\begingroup$ Ma (2020) showed that $I(\mathbf{X}) = \iint_{\mathbf{u}} c(\mathbf{u}) \ln c(\mathbf{u}) \, d\mathbf{u}$ where $c(\cdot)$ is a multivariate copula density, and $\mathbf{u}$ are the uniform marginals corresponding to $\mathbf{X}$ arxiv.org/abs/2005.14025 $\endgroup$ – develarist Dec 10 '20 at 14:33

To see where the results comes from using another approach, first recall that if $X,Y$ have joint distribution $p(X,Y)$ and marginals $p(X)$ and $p(Y)$, respectively, then the mutual information is the Kullback-Leibler (KL) divergence between the joint distribution and product of marginal distributions:

$$I(X:Y) = D_{\rm KL}(p(X,Y)\mid\mid p(X)p(Y)).$$

Next, for a pair of $n$-dimensional multivariate normal distributions with mean and covariance parameters $(\mu, \Sigma)$ and $(\tilde{\mu}, \tilde{\Sigma})$, respectively, the KL divergence is given by:

$$ \begin{aligned} &D_{\rm KL}((\mu,\Sigma) \mid\mid (\tilde{\mu}, \tilde{\Sigma})) \\ &= \frac{1}{2} \left[ \log\left( \mathrm{det}(\tilde{\Sigma})/ \mathrm{det}(\Sigma) \right) + \mathrm{tr}\left( \Sigma \tilde{\Sigma}^{-1} \right) - n + (\mu - \tilde{\mu})^\top\tilde{\Sigma}^{-1}(\mu - \tilde{\mu}) \right]. \end{aligned} $$

Suppose now that the joint distribution $p(X,Y)$ is multivariate normal with mean and covariance given by

$$ \mu = \begin{bmatrix} \mu_x \\ \mu_y \end{bmatrix}, \qquad \Sigma = \begin{bmatrix} \Sigma_x & \Sigma' \\ \Sigma' & \Sigma_y \end{bmatrix}. $$

The product of marginals $p(X)p(Y)$ is also multivariate normal, with mean and covariance given by

$$ \tilde{\mu} = \begin{bmatrix} \mu_x \\ \mu_y \end{bmatrix}, \qquad \tilde{\Sigma} = \begin{bmatrix} \Sigma_x & \mathbf{0} \\ \mathbf{0} & \Sigma_y \end{bmatrix}. $$

To complete the computation, we simplify the expression for $D_{\rm KL}$ between two multivariate normal distributions whose are parameters are $(\mu, \Sigma)$ and $(\tilde{\mu}, \tilde{\Sigma})$ as given above, by using the following facts:

  1. Since $\tilde{\Sigma}$ is a block matrix and $\Sigma_y$ is invertible, we have

$$\mathrm{det}(\tilde{\Sigma}) = \mathrm{det}(\Sigma_x - \mathbf{0}\Sigma_y^{-1}\mathbf{0})\mathrm{det}(\Sigma_y) = \mathrm{det}(\Sigma_x)(\Sigma_y). $$

  1. From the block structure of $\tilde{\Sigma}$, we have

$$ \tilde{\Sigma}^{-1} = \begin{bmatrix} \Sigma_x^{-1} & \mathbf{0} \\ \mathbf{0} & \Sigma_y^{-1} \end{bmatrix} $$ which implies that $$ \Sigma \tilde{\Sigma}^{-1} = \begin{bmatrix} \Sigma_x & \Sigma' \\ \Sigma' & \Sigma_y \end{bmatrix} \begin{bmatrix} \Sigma_x^{-1} & \mathbf{0} \\ \mathbf{0} & \Sigma_y^{-1} \end{bmatrix} = \begin{bmatrix} \Sigma_x\Sigma_x^{-1} & \Sigma'\Sigma_y \\ \Sigma'\Sigma_x & \Sigma_y^{-1} \end{bmatrix} = \begin{bmatrix} \mathbf{1}_{n_1} & \Sigma'\Sigma_y \\ \Sigma'\Sigma_x & \mathbf{1}_{n_2} \end{bmatrix} $$

where $n_1$ and $n_2$ are positive integers such that $n_1 + n_2 = n$. Thus, it follows that $$ \mathrm{tr}\left( \Sigma \tilde{\Sigma}^{-1} \right) - n = n - n = 0. $$

  1. Since $\mu = \tilde{\mu}$, the right-most term in the expression for $D_{\rm KL}$ is equal to zero:

$$ \frac{1}{2}(\mu - \tilde{\mu})^\top\tilde{\Sigma}^{-1}(\mu - \tilde{\mu}) = 0. $$

Using these facts, we have shown that

$$ I(X:Y) = D_{\rm KL}((\mu,\Sigma) \mid\mid (\tilde{\mu}, \tilde{\Sigma})) = \frac{1}{2} \left[ \log\left( \frac{\mathrm{det}(\Sigma_x)\mathrm{det}(\Sigma_y)}{\mathrm{det}(\Sigma)} \right) \right] $$

as desired.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.