3
$\begingroup$

KL-divergence is often defined as $$D_{KL}(P||Q) = E_{x\sim P} \left[ \log\left(\frac{P(X)}{Q(X)}\right) \right] = \int_{-\infty}^{\infty} P(x) \log\left(\frac{P(X)}{Q(X)}\right)dx$$

I don't quite understand the meaning of the notation of the second term and how you can get the third term from there. The fact that there is the subscript $x \sim P$ under the expectation seems to indicate that there are several random variables in the Expectation bracket and the subscript is here to clarify the variable with respect to which we're integrating. In such cases, the notation I'm used to is $E_X[h(X,Y)]$ (eg in this answer). But here, I'm confused by the fact that there seems to be a single random variable $X$ following two different distributions. So could this expression be rewritten with two separate random variables $X$ and $Y$ ?

Regarding the derivation of the third term, do you get it via the LOTUS, setting $g(X)$ as $\log\left(\frac{P(X)}{Q(X)}\right)$ ? I've already seen functions of random variables, but I find it strange that in this case, the $g$ function would the probability distributions $P(X)$ and $Q(X)$ of the random variable(s)... All this makes me think that I'm misunderstanding a basic notation. Can someone walk me through the explanation of the meaning of the second term and the derivation of the third one ?

$\endgroup$
1
  • 3
    $\begingroup$ 1) the notation $\mathbb{E}_X$ is insufficient because we have two candidate distributions for $X$: $P$ and $Q$. Here, we're making clear that the expectation is with $X\sim P$. 2) regarding "the $g$ function would the probability distributions $P(X)$ and $Q(X)$", it may help to be precise: $g$ is not "the probability distributions", in particular $g$ is the (logarithmic ratio of the) density functions of these distributions. $\endgroup$ Jan 10, 2023 at 13:49

1 Answer 1

7
$\begingroup$

There is only one random variable, $X$, involved, but two different distributions, $P$ and $Q$. So somehow the notation must specify under which distribution the expectation is to be calculated. The notation $x\sim P$ signifies that the distribution to be used for $X$ is $P$ (but you should have written $X\sim P$). See also Reference: who introduced the tilde "~" notation to mean "has probability distribution..."?

In the last integral, $P$ and $Q$ denotes the densities, It might be better to use $P, Q$ to symbolize the distributions, and $p, q$ for the corresponding densities. Also, be more consistent with the use of $X$ and $x$. So a better version of your formula is $$ D_{KL}(P||Q) = E_{X\sim P} \left[ \log\left(\frac{P(X)}{Q(X)}\right) \right] = \int_{-\infty}^{\infty} p(x) \log\left(\frac{p(x)}{q(x)}\right) \; dx $$ Here, in the second term, $X$ represents the random variable, while in the last integral $x$ is a dummy variable of integration. See also Intuition on the Kullback–Leibler (KL) Divergence for more on the meaning of this KL divergence, which might help to clarify the notation further.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks ! Clearing up that (i) a single random variable can be involved in two different distributions and (ii) the difference between distributions and densities, really pushed my understanding forward. $\endgroup$
    – Soltius
    Jan 24, 2023 at 10:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.