3
$\begingroup$

Mutual information is defined as the Kullback-Leibler divergence between a joint distribution and its marginals:

$$I(X,Y) = \mathrm{KL}(P(x,y)||P(x)P(y)) = \sum_{x,y}P(x,y)\ln\left(\frac{P(x,y)}{P(x)P(y)}\right)$$

Is there a name for the reverse Kullback-Leibler divergence:

$$\mathrm{KL}(P(x)P(y)||P(x,y)) = \sum_{x,y}P(x)P(y)\ln\left(\frac{P(x)P(y)}{P(x,y)}\right)$$

It shares some properties with the mutual information. E.g., it is zero if and only if $x,y$ are independent. Why is it less popular than mutual information?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

Note that the joint distribution may be supported on a strictly smaller subset than the product of marginals (consider for example a joint distribution supported on the diagonal, so that the product of marginals is supported everywhere). In that case mutual information is well-defined, but the reverse version is not.

Improve this answer
$\endgroup$
1
$\begingroup$

If you're just swapping the inputs $P||Q$ to $Q||P$, then this is called reverse KL-divergence. Unlike forward KL-divergence that you showed which has mean-seeking behavior, reverse-KL divergence has mode-seeking ability.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Yes, that's right. It's the reverse KL-divergence of the mutual information. But does it have a name? It shares some of the properties of the mutual information (e.g., it is zero if and only if $x,y$ are independent. $\endgroup$
    – a06e
    Commented Sep 17, 2020 at 14:03
  • $\begingroup$ It's name is reverse KL-divergence $\endgroup$
    – develarist
    Commented Sep 17, 2020 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.