# Which definition of Kullback-Leibler-Divergence for discrete & continuos variables makes sense?

Motivation In variational inference, one tries to approximate a posterior density $$p(\theta|y)$$ by another,easier, density $$q(\theta)$$ in terms of the Kullback-Leibler-Divergence $$D_{KL}(q(\theta)||p(\theta|y)) = \int q(\theta) \ln \frac{q(\theta)}{p(\theta|y)} d\theta$$

This is fine as long as both densities are absolutely continuos w.r.t. some measure. Now assume the case that $$q$$ is a point mass $$\delta_x$$. Denote by $$P$$,$$Q$$, the respective measures.

Question

In here and here (also check this question of mine) , I have found that a natural extension of the Kullback-Leibler-Divergence in such cases is to take the densities with respect to the measure $$P+Q$$ and use the same definition as above. This will yield the value $$\infty$$ in the above case of a pointmass and a continuos density.

On the other hand, in the case of continuos densities, one has the decomposition $$D_{KL}(q||p) = -\mathbb{E}_Q[\ln p] - \mathbb{H}[Q]$$

If one would apply this defintion now to the above case, this would result in $$- \ln p(x|y)$$, i.e. minimizing the Divergence would correspond to a maximum-a-posteriori estimate.

So my question is: Am I missing something here, i.e. are both definitions actually equivalent, which I am not seeing? If not, which definition makes more sense, e.g. which one would be continuos in some sense?

Edit:

Some additional thoughts: Both definitions to me have some intuitive value: If one interprets the KL-Divergence as a quantity indicating similiarity of probability measures, the first definition (yielding $$\infty$$) tells us that there is not any particular point measure that is 'closer' to a given continuos distribution, they are all equally, 'infinitely', dissimilar to it.

On the other hand, the second definition would tell us that among all point masses, those that match a mode of the continuos distribution are the most similar to it, which also has intuitive appeal.

In terms of continuity, it seems that the second definition will not be continuos

Consider $$p(\theta|y) = \mathcal{N}(0,1)$$, $$q_n(\theta) = \mathcal{N}(0,\frac{1}{n})$$, such that $$Q_n \overset{D}{\rightarrow} \delta_0$$ Then we have

$$D_{KL}(q_n||p) = -\mathbb{E}_{Q_n}[\ln p] - \mathbb{H}[Q_n] \\ = \frac{1}{2n} -\frac{1}{2} \ln \frac{1}{n} + C \overset{n \to \infty}{\rightarrow} \infty$$

Where the main point here seems to be that $$\mathbb{H}$$, when applied to a continuos distribution, i.e. the differential entropy, does not converge to the Shannon entropy of the limiting point mass, is that correct?

Edit:

Here I want to include my understanding of why the definition w.r.t. $$P+Q$$ yields $$\infty$$. Recall that for a dirac measure $$Q = \delta_x$$ we have

$$\int\limits_{A} f dQ = f(x) \mathbb{I}[x \in A],$$ which we will repeatedly use. Also recall that the Radon-Nikodyn-density of a measure $$M_1 << M_2$$ is a nonnegative measurable function $$\frac{dM_1}{dM_2}$$ such that $$M_1(A) = \int \limits_{A}\frac{dM_1}{dM_2} dM_2$$ for all measurable sets $$A$$.

We have that

$$\frac{dP}{dP + dQ} = 1 - \mathbb{I}[\theta =x],$$

since $$\int\limits_{A} 1 - \mathbb{I}[\theta =x] dP + dQ = \\ \int\limits_{A} 1 - \mathbb{I}[\theta =x] dP + \int\limits_{A} 1 - \mathbb{I}[\theta =x] dQ \\ = \int\limits_{A} 1 - \mathbb{I}[\theta =x] dP + \mathbb{I}[x \in A](1 - \mathbb{I}[x=x]) \\ = \int\limits_{A} 1 - \mathbb{I}[\theta =x] dP + 0 = \int\limits_{A} 1 - \mathbb{I}[\theta =x] dP .$$

We have $$\frac{dQ}{dP + dQ} = \mathbb{I}[\theta =x],$$

since

$$\int\limits_{A} \mathbb{I}[\theta =x] dP + dQ = \\ \int\limits_{A} \mathbb{I}[\theta =x] dP + \int\limits_{A} \mathbb{I}[\theta =x] dQ \\ = 0 + \mathbb{I}[x \in A] \mathbb{I}[x= x] =\mathbb{I}[x \in A] = \int\limits_{A} dQ$$

Now, the definition of the Kullback-Leibler-Divergence with respect to $$P+Q$$, that can be found in the linked post, is given by

$$D_{KL}(Q||P) = \int \frac{dQ}{dP + dQ} \frac{\frac{dQ}{dP + dQ}}{\frac{dP}{dP + dQ}} d P + Q$$

Plugging in the above obtained Radon-Nikodyn derivatives, we get

$$D_{KL}(Q||P) = \int \mathbb{I}[\theta = x] \ln \frac{\mathbb{I}[\theta = x]}{1- \mathbb{I}[\theta = x]} d P + Q \\ = \int \mathbb{I}[\theta = x] \ln \frac{\mathbb{I}[\theta = x]}{1- \mathbb{I}[\theta = x]} d P + \int \mathbb{I}[\theta = x] \ln \frac{\mathbb{I}[\theta = x]}{1- \mathbb{I}[\theta = x]} dQ \\$$

Now recall that $$P$$ is absolutely continuos w.r.t. the Lebesque measure, and we have $$\mathbb{I}[\theta = x] \ln \frac{\mathbb{I}[\theta = x]}{1- \mathbb{I}[\theta = x]} = 0 \ \text{Lebesque a.e.}$$

such that the first term above evaluates to 0.

For the second term we get

$$\int \mathbb{I}[\theta = x] \ln \frac{\mathbb{I}[\theta = x]}{1- \mathbb{I}[\theta = x]} dQ \\ = \mathbb{I}[x = x] \ln \frac{\mathbb{I}[x = x]}{1- \mathbb{I}[x = x]} \\ = \mathbb{I}[x = x] \ln \mathbb{I}[x = x] - \mathbb{I}[x = x] \ln( 1- \mathbb{I}[x = x]) \\ = 1 \ln 1 - 1 \ln 0 = \infty$$

• Potentially related question about mixed distributions: math.stackexchange.com/q/3694054/672549 – Dave Jun 12 '20 at 13:54
• How did you determine that "this will yield the value $\infty$"? The whole point of using $P+Q$ as a reference measure is that both $P$ and $Q$ are a.c. with respect to it, meaning $D_{KL}$ ought to be finite. – whuber Jun 12 '20 at 15:10
• @whuber To be honest, I just copied it from the first question I linked to. I have to say that my measure theoretic foundations are not the best, so maybe that is my problem here. I thought the derivation is like this: $\frac{dP}{dQ + dP} = 1 - \mathbb{I}[\theta = x]$, $\frac{dQ}{dQ + dP} = \mathbb{I}[\theta = x]$. Hence, one has that $D_{KL} = \int \mathbb{I}[\theta = x] \ln \frac{ \mathbb{I}[\theta = x]}{1 - \mathbb{I}[\theta = x]} dP + \int \mathbb{I}[\theta = x] \ln \frac{ \mathbb{I}[\theta = x]}{1 - \mathbb{I}[\theta = x]} dQ$ – a_student Jun 12 '20 at 16:43
• $= 0 + \mathbb{I}[x = x] \ln \frac{\mathbb{I}[x = x]}{1 -\mathbb{I}[x = x] } = - \ln 0 = \infty$ - Is that not correct? What would the correct result be? – a_student Jun 12 '20 at 16:46
• @whuber Sorry to bother you, but did you have time to check the derivation I posted? I am still confused by this question... – a_student Jul 10 '20 at 16:16

## 1 Answer

Mine is not an answer, but just a try to make your correct procedure clearer.

Using Lebesgue theorem, you can consider the absolute continuous measure $$P+Q$$ as made of an absolute continuous measure $$P$$ and a singular $$Q=\delta_x$$.

To use the Radon-Nikodyn theorem, both the measures must be absolute continuous wrt $$P+Q$$ and $$\sigma$$-finite, which is true in your case.

Denoted $$(\mathbb{X}_P, \mathcal{A}_P)$$ and $$(\mathbb{X}_Q, \mathcal{A}_Q)$$ the disjoint measurable spaces with measures $$P$$ and $$Q$$, respectively, their union $$(\mathbb{X}, \mathcal{A})$$ is a measurable space with measure equal to $$P+Q$$.

As described here, given the densities $$f$$ and $$g$$ of $$P$$ and $$Q$$ wrt the measure $$P+Q$$ for the measurable space $$(\mathbb{X},\mathcal{A})$$ (these are given by Radon-Nikodyn theorem), one can write the KL:

$$KL(P||Q)=\int_{\mathbb{X}} f\ln\frac{f}{g}\,d(P+Q)$$

For Radon-Nikodyn theorem, the density of $$Q$$ wrt $$P+Q$$ is the Dirac $$\delta_x=\mathbb{I}[\theta=x]$$, while the density of $$P$$ wrt $$P+Q$$ is $$1-\mathbb{I}[\theta=x]$$. Intuitively, this means that, in $$\mathbb{X}$$, $$P$$ is equal to itself in $$\mathbb{X}_P$$ as like as $$Q$$ in $$\mathbb{X}_Q$$.

These can be seen in the same way as you did (EDIT: fixed a mistake):

$$\int_{\mathbb{X}}1-\mathbb{I}[\theta=x]\,d(P+Q)=\\ \int_{\mathbb{X}_P}1-\mathbb{I}[\theta=x]\,dP + \int_{\mathbb{X}_Q}1-\mathbb{I}[\theta=x]\,dQ=\\ \int_{\mathbb{X}_P}1-\mathbb{I}[\theta=x]\,dP+0=\\ \int_{\mathbb{X}_P}dP - \int_{\mathbb{X}_P}\mathbb{I}[\theta=x]\,dP=\\ \int_{\mathbb{X}_P}dP - 0 =\int_{\mathbb{X}}dP$$

given that $$\int_{\mathbb{X}_Q} dP=0$$, and

$$\int_{\mathbb{X}}\mathbb{I}[\theta=x]\,d(P+Q)=\\ \int_{\mathbb{X}_P}\mathbb{I}[\theta=x]\,dP + \int_{\mathbb{X}_Q}\mathbb{I}[\theta=x]\,dQ=\\ 0+\int_{\mathbb{X}_Q}\mathbb{I}[\theta=x]\,dQ=\\ \int_{\mathbb{X}_Q}dQ=\int_{\mathbb{X}}dQ$$

given that $$\int_{\mathbb{X}_P} dQ=0$$.

The formulation of $$KL(Q||P)$$ in terms of transformed densities wrt to $$P+Q$$ is correct (using the densities of $$P$$ and $$Q$$ wrt $$P+Q$$):

$$KL(Q||P)=\int_\mathbb{X} \ln\frac{dQ}{dP}\,dQ=\\ \int_\mathbb{X} \ln\left(\frac{dQ/d(P+Q)}{dP/d(P+Q)}\right)\,\frac{dQ}{d(P+Q)}d(P+Q)$$

Then, substituting the analytical expressions of $$f$$ and $$g$$ you get the result.

• Thank you for these acclarations! Do you have any thoughts on the question? I added some of my thoughts on the continuity of the definitions above. – a_student Jul 11 '20 at 13:48
• Yes. Entropy of Dirac $\delta$ diverges. You can see the same result using a different formulation here math.stackexchange.com/a/626696 – user289381 Jul 11 '20 at 14:30