The Kullback–Leibler (KL) divergence of two continuous distributions $P(x)$ and $Q(x)$ is defined as

$$D_{KL}(P \mid\mid Q) = \int_{X} P(x) \log{\left[\frac{P(x)}{Q(x)}\right]} dx$$

How can one compute the KL divergence when $Q$ is the Dirac delta function $Q(x)=\delta(x-x_0)$?

Of course, we can expand the logarithm above to

$$D_{KL}(P \mid\mid Q) = \int_{X} P(x) \log{\left[P(x)\right]} dx - \int_{X} P(x) \log{\left[\delta(x-x_0)\right]} dx$$

How is the integral on the right evaluated, given that the logarithm approaches $-\infty$ for all $x \ne x_0$ and $\infty$ for $x=x_0$?

  • 2
    $\begingroup$ KL divergence requires that $P\ll Q$, which translates to $P$ assigning 0 measure to sets $A$ whenever $Q(A)=0$. In this case $Q(A)=0$ whenever $x_0\notin A$. So the only sensible $P$ here would be $P=Q$, which is silly. $\endgroup$ – Alex R. Jul 17 '17 at 23:00
  • $\begingroup$ @AlexR., Thanks for your reply. Could you explain what is meant by $P \ll Q$? Does this mean that $P(x) \ll Q(x)$ for all $x$? That doesn't seem possible for proper distributions. Or does it mean that the maximum value of $P(x)$ is much smaller than the maximum value of $Q(x)$, i.e., that $Q(x)$ is more peaked? Because that would seem to follow in this case. $\endgroup$ – saxen Jul 17 '17 at 23:27
  • 1
    $\begingroup$ @AlexR., additionally, my understanding is that $D_{KL}(P \mid\mid Q)$ and $D_{KL}(Q \mid\mid P)$ can both exist for the same $P$ and $Q$. How can it be that $P \ll Q$ and $Q \ll P$ can both be true? $\endgroup$ – saxen Jul 17 '17 at 23:30
  • $\begingroup$ $P\ll Q$ means $P$ is absolutely continuous with respect to $Q$: mathworld.wolfram.com/AbsolutelyContinuous.html As an example, if you have two distributions with non-zero density functions, then they are absolutely continuous with respect to eachother (because the only sets which they assign 0 mass to are singletons). $\endgroup$ – Alex R. Jul 17 '17 at 23:49
  • $\begingroup$ @AlexR., Ah, that makes more sense. So that's why, for the example above, if $Q$ is the Dirac delta, we could compute $D_{KL}(Q \mid\mid P)$, but not $D_{KL}(P \mid\mid Q)$. $\endgroup$ – saxen Jul 18 '17 at 1:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.