• Suppose $P$ and $Q$ are two distributions on a space ${\cal H}$ (could be a subset of an infinite dimensional function space) with p.d.fs denoted by the same letter then one can define the $KL$ divergence between these distributions as, $$KL (Q || P) = \mathbb{E}_{h \sim Q} [ \log \frac{Q(h)}{P(h)} ] = \int_{{\cal H}} \log \frac{Q(h)}{P(h)} Q(h) d\mu(h) $$

  • Suppose $P$ and $Q$ are two probability measures on ${\cal H}$ s.t $Q$ is absolutely continuous w.r.t $P$ so that the Radon-Nikodym derivative exists. Suppose for any non-negative random variable with finite mean we define $Ent[Z] = \mathbb{E} [ Z \log \frac{Z}{\mathbb{E}[Z]} ]$. Then one can define the $KL$ divergence between these measures as, $$KL (Q || P) = Ent [ \frac{dQ}{dP} ]$$

My question is : When are these two notions the same?

In both cases we need ${\cal H}$ to be a measure space and I am calling the measure defined on it as $\mu$. Kindly correct if I am writing anything wrong!

In the first equality I guess it is always implicit that one is having two continuous random variables valued in the measure space ${\cal H}$ whose probability distributions are $P$ and $Q$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.