Can the relative entropy (Kullback Leibler divergence) between multivariate distributions be decomposed into relative entropies of the different variables plus some measure of dependence between the variables (something like the mutual information)?

If $X,Y$ are random variables, can we obtain this:

$$D_{KL}\big( p(X,Y) \parallel q(X,Y) \big) = D_{KL}\big( p(X) \parallel q(X) \big) + D_{KL}\big( p(Y) \parallel q(Y) \big)$$ $$+ \ (\text{dependence between } X,Y)$$


Yes, the KL divergence between joint distributions $p_{XY}$ and $q_{XY}$ can be expressed in terms of the KL divergence between the marginal distributions ($p_X, p_Y, q_X, q_Y$), plus terms that measure the dependence between $X$ and $Y$.

The general case

First, without assuming any particular form for the distributions, write the KL divergence between $p_{XY}$ and $q_{XY}$ as the difference of the cross entropy and the joint entropy:

$$D_{KL}(p_{XY} \parallel q_{XY}) = H(p_{XY}, q_{XY}) - H(p_{XY}) \tag{1}$$

The mutual information between $X$ and $Y$ (for distribution $p_{XY}$) is equal to the sum of the marginal entropies minus the joint entropy:

$$I(p_{XY}) = H(p_X) + H(p_Y) - H(p_{XY}) \tag{2}$$

Solving (2) for the joint entropy and plugging it into (1) gives:

$$D_{KL}(p_{XY} \parallel q_{XY}) = H(p_{XY}, q_{XY}) + I(p_{XY}) - H(p_X) - H(p_Y) \tag{3}$$

Analogous to (1), we know that:

$$D_{KL}(p_X \parallel q_X) = H(p_X, q_X) - H(p_X) \tag{4}$$ $$D_{KL}(p_Y \parallel q_Y) = H(p_Y, q_Y) - H(p_Y) \tag{5}$$

Solving (4) and (5) for the marginal entropies and plugging them into (3) gives:

$$D_{KL}(p_{XY} \parallel q_{XY}) \ = \ D_{KL}(p_X \parallel q_X) + D_{KL}(p_Y \parallel q_Y) + I(p_{XY}) - J(p_{XY}, q_{XY}) \tag{6}$$

with $J$ defined as:

$$\begin{array}{ccl} J(p_{XY}, q_{XY}) & = & H(p_X, q_X) + H(p_Y, q_Y) - H(p_{XY}, q_{XY}) \\ & = & E_{p_{XY}} \left[ \log \frac{q_{XY}}{q_X q_Y} \right] \\ \end{array} \tag{7}$$

Notice the similarity of $J$ to $I(q_{XY})$, the mutual information between $X$ and $Y$ for distribution $q_{XY}$. The difference is that, for $J$, the expectation is taken with respect to $p_{XY}$ instead of $q_{XY}$. The value depends on both the dependence between $X$ and $Y$ (under $q_{XY}$) and the similarity between $p_{XY}$ and $q_{XY}$.

Assuming independence for $q$

Suppose $p_{XY}$ may take any form, but $q_{XY}(x,y) = q_X(x) q_Y(y)$. That is, the joint distribution factorizes as the product of the marginals, so $X$ and $Y$ are independent. In this case, we can see from (7) that $J(p_{XY}, q_{XY}) = E_{p_{XY}}[\log 1] = 0$. Plugging this into (6):

$$D_{KL}(p_{XY} \parallel q_{XY}) = D_{KL}(p_X \parallel q_X) + D_{KL}(p_Y \parallel q_Y) + I(p_{XY}) \tag{8}$$

Assuming independence for both $p$ and $q$

Now, suppose $p_{XY}(x,y) = p_X(x) p_Y(y)$ as well, so $X$ and $Y$ are independent for both distributions. In this case, we know that the mutual information $I(p_{XY}) = 0$. Plugging this into (8):

$$D_{KL}(p_{XY} \parallel q_{XY}) = D_{KL}(p_X \parallel q_X) + D_{KL}(p_Y \parallel q_Y) \tag{9}$$

As a sanity check, this result can also be obtained by plugging the factorized forms of $p_{XY}$ and $q_{XY}$ directly into the standard formula for KL divergence.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.