KL divergence between which distributions could be infinity

Question

I know that KL divergence measures difference between two probability distributions. My doubt is for which of the distributions it could become Infinity, putting it in another way, P(x) has to produce the values that Q(x) cannot.

Or Q(x) has to be 0 where P(x) does not, because KL divergence is given as below:

$$\int_{-\infty}^{\infty}\{\log\frac{P(x)}{Q(x)}\}P(x)dx$$

Can any one tell me by choosing which probability distributions for P(x) and Q(x) we can achieve KL divergence as Infinity.

I think choosing P(x) as uniform distribution and Q(x) as Gaussian distribution could give Infinity, am I right?

Answer 1

What happens to $D_{KL}(p \parallel q)$ when $p(x)$ and/or $q(x)$ is zero? In a strict sense, the log of zero is undefined because there's no value of $x$ such that $e^x = 0$. But, the definition of KL divergence uses the following conventions (see Cover and Thomas, Elements of Information Theory):

$$0 \log \frac{0}{0} = 0, \quad 0 \log \frac{0}{q(x)} = 0, \quad p(x) \log \frac{p(x)}{0} = \infty$$

This implies that KL divergence is infinite if there exists an $x$ where $p(x) > 0$ and $q(x) = 0$. Conversely, $p(x)$ being zero doesn't produce infinity, whether or not $q(x)$ is also zero. Another way of saying this is that KL divergence is finite only if the support of $p$ is contained within the support of $q$. However, note that KL divergence can be infinite even if $p(x)$ and $q(x)$ are nonzero for all $x$ (see here for an example).

Regarding your example, this means that KL divergence will be infinite if $p$ is Gaussian and $q$ is uniform, but not the other way around.