1
$\begingroup$

from the formula of KL divergence $$ KL(P||Q) = \int_{-\infty}^{\infty} p(x) \log \left(\frac{p(x)}{q(x)}\right)\operatorname{d}x$$ I feel the $P$ and $Q$ has the same random variable, but, the distribution associated with a random variable should be unique, so how could we have two separate distributions with the same random variable? I can think $x$ is just dummy variable at here but again, how could this be valid from the definition?

$\endgroup$

1 Answer 1

1
$\begingroup$

When considering $$KL(P||Q) = \int_{-\infty}^{\infty} p(x) \log \left(\frac{p(x)}{q(x)}\right)~\text{d}x$$ it can be expressed as $$KL(P||Q) = \mathbb E_p\left[\log \left(\frac{p(X)}{q(X)}\right)\right]$$ meaning it is the expectation of a transform of the random variable $X$ with density $p$ (or distribution $P$). There is therefore no assumption of $X$ being distributed from both $P$ and $Q$. Note however that it can equally be written as $$KL(P||Q) = \int_{-\infty}^{\infty} q(x)~\frac{p(x)}{q(x)}~\log \left(\frac{p(x)}{q(x)}\right)~\text{d}x=\mathbb E_q\left[\frac{p(X)}{q(X)}~\log \left(\frac{p(X)}{q(X)}\right)\right]$$meaning that it can also be expressed as an integral under $Q$ (or any other distribution following the same reasoning).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.