2
$\begingroup$

We know KL-divergence is sometimes expressed like this:

KL-divergence

which shows it's capturing the deviation between the joint distribution of X and Y, and the product of marginals for X and Y. This suggests KL-divergence is simply the multiplication rule for independent events, reformulated in terms of entropy. In other words, if the joint fails to align with the product of marginals then it's expected the variables have some dependence structure.

We also see KL-divergence expressed like this:

KL-divergence as well

How does this second expression relate to the first? I cannot see where the joint is being calculated, or the product of marginals for that matter.

Improve this question
$\endgroup$
13
  • 1
    $\begingroup$ The first expression is the mutual information between $X$ and $Y$, which is only a special instance of a KL divergence. Not all KL divergences take the form of a mutual information. $\endgroup$
    – πr8
    Aug 24, 2020 at 17:53
  • $\begingroup$ @πr8 But we should still be able to understand the first expression in terms of the second. $\endgroup$
    – Cybernetic
    Aug 24, 2020 at 17:55
  • 1
    $\begingroup$ Yes, the first expression is the KL divergence between the joint $P(x, y)$ and the product of the marginals $P(x) \cdot P(y)$. The first expression is a special case of the second. $\endgroup$
    – πr8
    Aug 24, 2020 at 17:59
  • $\begingroup$ that is, you can write $I(X, Y) = \sum_{x, y} P (x, y) \log \frac{ P (x, y) }{ P(x) \cdot P(y) }.$ is this what you mean? $\endgroup$
    – πr8
    Aug 24, 2020 at 18:00
  • 1
    $\begingroup$ what do you mean by "treat both a joint and a product of marginals as though they were regular, single distributions"? they are both valid distributions over $(x, y)$ space, if that answers your question. $\endgroup$
    – πr8
    Aug 24, 2020 at 18:39

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.