2
$\begingroup$

I want to calculate KL Divergence between a normal and an exponential r.v. i.e. $$D(P||Q) = ?\\ \;\; P=N(\mu,\sigma), \;\; Q=exp(\lambda)$$ My problem is that in this case the domains of the distributions are different - the domain of $P$ is $x\in R$ and the domain of Q is $x \in [0,\infty )$. Which domain should I integrate over? If this is the domain of $P$ the value of $\log(Q(x)/P(x))$ is not defined.

Let's say we use want to calculate the KL Divergence for $\mu = 1, \sigma = 2 ,\lambda =1$ what will be the result? I can calculate $D(Q||P)$ but it is not the same.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

In this case the KL-divergence $D(P||Q)$ is indeed infinity. To have a well defined (not infinity) KL-divergence, we need support$(P)\subseteq$support$(Q)$. Here "support" means whereever the probability is non-zero. See discussion here (page 3).

Sometimes researchers choose to use the earth-mover distance / sinkhorn distance to avoid such problems.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Note that this implies that if we calculated $D(Q||P)$ we would not have a problem, although how meaningful the result would be is open to question! $\endgroup$
    – jbowman
    Sep 29, 2018 at 18:45
  • $\begingroup$ If support(P) = support(Q) we can still have infinity KL-divergence, see here $\endgroup$
    – Rodvi
    May 22, 2020 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.