# Kullback-Leibler divergence between marginals and joint distribution? (It's not mutual information)

Mutual information is defined as the Kullback-Leibler divergence between a joint distribution and its marginals:

$$I(X,Y) = \mathrm{KL}(P(x,y)||P(x)P(y)) = \sum_{x,y}P(x,y)\ln\left(\frac{P(x,y)}{P(x)P(y)}\right)$$

Is there a name for the reverse Kullback-Leibler divergence:

$$\mathrm{KL}(P(x)P(y)||P(x,y)) = \sum_{x,y}P(x)P(y)\ln\left(\frac{P(x)P(y)}{P(x,y)}\right)$$

It shares some properties with the mutual information. E.g., it is zero if and only if $$x,y$$ are independent. Why is it less popular than mutual information?

If you're just swapping the inputs $$P||Q$$ to $$Q||P$$, then this is called reverse KL-divergence. Unlike forward KL-divergence that you showed which has mean-seeking behavior, reverse-KL divergence has mode-seeking ability.
• Yes, that's right. It's the reverse KL-divergence of the mutual information. But does it have a name? It shares some of the properties of the mutual information (e.g., it is zero if and only if $x,y$ are independent. – becko Sep 17 '20 at 14:03