8
$\begingroup$

I know that KL divergence measures difference between two probability distributions. My doubt is for which of the distributions it could become Infinity, putting it in another way, P(x) has to produce the values that Q(x) cannot.

Or Q(x) has to be 0 where P(x) does not, because KL divergence is given as below:

$$\int_{-\infty}^{\infty}\{\log\frac{P(x)}{Q(x)}\}P(x)dx$$

Can any one tell me by choosing which probability distributions for P(x) and Q(x) we can achieve KL divergence as Infinity.

I think choosing P(x) as uniform distribution and Q(x) as Gaussian distribution could give Infinity, am I right?

$\endgroup$

1 Answer 1

16
$\begingroup$

What happens to $D_{KL}(p \parallel q)$ when $p(x)$ and/or $q(x)$ is zero? In a strict sense, the log of zero is undefined because there's no value of $x$ such that $e^x = 0$. But, the definition of KL divergence uses the following conventions (see Cover and Thomas, Elements of Information Theory):

$$0 \log \frac{0}{0} = 0, \quad 0 \log \frac{0}{q(x)} = 0, \quad p(x) \log \frac{p(x)}{0} = \infty$$

This implies that KL divergence is infinite if there exists an $x$ where $p(x) > 0$ and $q(x) = 0$. Conversely, $p(x)$ being zero doesn't produce infinity, whether or not $q(x)$ is also zero. Another way of saying this is that KL divergence is finite only if the support of $p$ is contained within the support of $q$. However, note that KL divergence can be infinite even if $p(x)$ and $q(x)$ are nonzero for all $x$ (see here for an example).

Regarding your example, this means that KL divergence will be infinite if $p$ is Gaussian and $q$ is uniform, but not the other way around.

$\endgroup$
2
  • $\begingroup$ You said "KL divergence is finite only if the support of $p$ is contained within the support of $q$". Why did you use the word only? If by "contained within" you mean $\subset$ then there is an obvious case $p \equiv q$ (then support(p)=support(q)) when KL=0 (finite). And If by "contained within" you mean $\subseteq$ then your claim is false (see your own hyperlink above). $\endgroup$
    – Rodvi
    Commented May 22, 2020 at 14:10
  • $\begingroup$ @Rodvi Please note the answer states "only if" (which is correct here), not "if and only if" (which would be incorrect). To unpack the statement: If $\operatorname{support}(p) \nsubseteq \operatorname{support}(q)$ then $D_{KL}(p \parallel q) = \infty$. Otherwise, if $\operatorname{support}(p) \subseteq \operatorname{support}(q)$, then $D_{KL}(p \parallel q)$ could be finite or infinite. $\endgroup$
    – user20160
    Commented Sep 25, 2020 at 18:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.