3
$\begingroup$

I want to calculate KL Divergence between a normal and an exponential r.v. i.e. $$D(P||Q) = ?\\ \;\; P=N(\mu,\sigma), \;\; Q=exp(\lambda)$$ My problem is that in this case the domains of the distributions are different - the domain of $P$ is $x\in R$ and the domain of Q is $x \in [0,\infty )$. Which domain should I integrate over? If this is the domain of $P$ the value of $\log(Q(x)/P(x))$ is not defined.

Let's say we use want to calculate the KL Divergence for $\mu = 1, \sigma = 2 ,\lambda =1$ what will be the result? I can calculate $D(Q||P)$ but it is not the same.

$\endgroup$

1 Answer 1

4
$\begingroup$

In this case the KL-divergence $D(P||Q)$ is indeed infinity. To have a well defined (not infinity) KL-divergence, we need support$(P)\subseteq$support$(Q)$. Here "support" means whereever the probability is non-zero. See discussion here (page 3).

Sometimes researchers choose to use the earth-mover distance / sinkhorn distance to avoid such problems.

$\endgroup$
2
  • 1
    $\begingroup$ Note that this implies that if we calculated $D(Q||P)$ we would not have a problem, although how meaningful the result would be is open to question! $\endgroup$
    – jbowman
    Commented Sep 29, 2018 at 18:45
  • $\begingroup$ If support(P) = support(Q) we can still have infinity KL-divergence, see here $\endgroup$
    – Rodvi
    Commented May 22, 2020 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.