I want to calculate KL Divergence between a normal and an exponential r.v. i.e. $$D(P||Q) = ?\\ \;\; P=N(\mu,\sigma), \;\; Q=exp(\lambda)$$ My problem is that in this case the domains of the distributions are different - the domain of $P$ is $x\in R$ and the domain of Q is $x \in [0,\infty )$. Which domain should I integrate over? If this is the domain of $P$ the value of $\log(Q(x)/P(x))$ is not defined.

Let's say we use want to calculate the KL Divergence for $\mu = 1, \sigma = 2 ,\lambda =1$ what will be the result? I can calculate $D(Q||P)$ but it is not the same.


In this case the KL-divergence $D(P||Q)$ is indeed infinity. To have a well defined (not infinity) KL-divergence, we need support$(P)\subseteq$support$(Q)$. Here "support" means whereever the probability is non-zero. See discussion here (page 3).

Sometimes researchers choose to use the earth-mover distance / sinkhorn distance to avoid such problems.

  • $\begingroup$ Note that this implies that if we calculated $D(Q||P)$ we would not have a problem, although how meaningful the result would be is open to question! $\endgroup$ – jbowman Sep 29 '18 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.