# Kullback-Leibler divergence between marginals and joint distribution? (It's not mutual information)

Mutual information is defined as the Kullback-Leibler divergence between a joint distribution and its marginals:

$$I(X,Y) = \mathrm{KL}(P(x,y)||P(x)P(y)) = \sum_{x,y}P(x,y)\ln\left(\frac{P(x,y)}{P(x)P(y)}\right)$$

Is there a name for the reverse Kullback-Leibler divergence:

$$\mathrm{KL}(P(x)P(y)||P(x,y)) = \sum_{x,y}P(x)P(y)\ln\left(\frac{P(x)P(y)}{P(x,y)}\right)$$

It shares some properties with the mutual information. E.g., it is zero if and only if $$x,y$$ are independent. Why is it less popular than mutual information?

## 2 Answers

Note that the joint distribution may be supported on a strictly smaller subset than the product of marginals (consider for example a joint distribution supported on the diagonal, so that the product of marginals is supported everywhere). In that case mutual information is well-defined, but the reverse version is not.

If you're just swapping the inputs $$P||Q$$ to $$Q||P$$, then this is called reverse KL-divergence. Unlike forward KL-divergence that you showed which has mean-seeking behavior, reverse-KL divergence has mode-seeking ability.

• Yes, that's right. It's the reverse KL-divergence of the mutual information. But does it have a name? It shares some of the properties of the mutual information (e.g., it is zero if and only if $x,y$ are independent. – becko Sep 17 '20 at 14:03
• It's name is reverse KL-divergence – develarist Sep 17 '20 at 14:12