Joint entropy of multivariate normal distribution less than individual entropy under high correlation

Question

Suppose we are calculating the joint entropy of a multivariate normal distribution with covariance matrix [1,0.99;0.99,1]. From the analytical solution, the joint entropy is 1.27; However, the marginal entropy for either random variable is 2.05 (i.e., when covariance matrix = [1]). I am wondering how can the joint entropy be less than the marginal entropy, any explanations?

Answer 1

This is possible in the case of continuous random variables, and follows from the fact that differential entropy can be negative.

First, recall that the joint entropy is related to the marginal and conditional entropies as:

$$\begin{array}{ccl} H(X,Y) & = & H(X) + H(Y \mid X) \\ & = & H(Y) + H(X \mid Y) \end{array} \tag{1}$$

For discrete random variables, entropy is always non-negative. Because the conditional entropies are non-negative, equation $(1)$ implies that the joint entropy is greater than or equal to both of the marginal entropies:

$$H(X,Y) \ge \max \{ H(X), H(Y) \}$$

But, this is not the case for continuous random variables, where entropy can be negative. In particular, equation $(1)$ implies that the joint entropy is less than the marginal entropy if the conditional entropy is negative. In your example, the conditional entropy is $H(Y \mid X) = H(X \mid Y) \approx -.78 \text{ bits}$

Answer 2

Recall that entropy is a measure for the uncertainty of a probability distribution. Distributions that are more uncertain and thus, observations from that particular distributions are harder to predict will exhibit higher entropy.

In the multivariate case, the imposed dependence structure reduces the unpredictability. To better understand this, it's convenient to write the joint entropy link this:

$H(X,Y)=H(X)+H(Y)-I(X,Y)$

Hereby denotes $I(X,Y)$ the mutual information of $X$ and $Y$, i.e. which part of $X$ may be explained by $Y$ and vice versa. In the case of independence, $I(X,Y)=0$. Therefore, for a given entropy of the margins, the joint entropy attains its maximum if the margins are independent (in the multivariate gaussian case, this coincides with $cor(X,Y)=0$). It's actually possible to relate the mutual information of two random variables to the dependence structure of those, via copula entropy.

Intuitively, you can think about it the following way. Assume that body weight and height are highly correlated: You wouldn't bet that you'd find a 1.90 meter person with a weight of 45 kg, whereas it could happen if both variables where independent. As a consequence, the dependence leads to a better predictability and less uncertainty in your multivariate distribution.