Which definition of Kullback-Leibler-Divergence for discrete & continuos variables makes sense?

Question

Motivation In variational inference, one tries to approximate a posterior density $p(\theta|y)$ by another,easier, density $q(\theta)$ in terms of the Kullback-Leibler-Divergence $$D_{KL}(q(\theta)||p(\theta|y)) = \int q(\theta) \ln \frac{q(\theta)}{p(\theta|y)} d\theta$$

This is fine as long as both densities are absolutely continuos w.r.t. some measure. Now assume the case that $q$ is a point mass $\delta_x$. Denote by $P$,$Q$, the respective measures.

Question

In here and here (also check this question of mine) , I have found that a natural extension of the Kullback-Leibler-Divergence in such cases is to take the densities with respect to the measure $P+Q$ and use the same definition as above. This will yield the value $\infty$ in the above case of a pointmass and a continuos density.

On the other hand, in the case of continuos densities, one has the decomposition $$D_{KL}(q||p) = -\mathbb{E}_Q[\ln p] - \mathbb{H}[Q]$$

If one would apply this defintion now to the above case, this would result in $- \ln p(x|y)$, i.e. minimizing the Divergence would correspond to a maximum-a-posteriori estimate.

So my question is: Am I missing something here, i.e. are both definitions actually equivalent, which I am not seeing? If not, which definition makes more sense, e.g. which one would be continuos in some sense?

---

Edit:

Some additional thoughts: Both definitions to me have some intuitive value: If one interprets the KL-Divergence as a quantity indicating similiarity of probability measures, the first definition (yielding $\infty$) tells us that there is not any particular point measure that is 'closer' to a given continuos distribution, they are all equally, 'infinitely', dissimilar to it.

On the other hand, the second definition would tell us that among all point masses, those that match a mode of the continuos distribution are the most similar to it, which also has intuitive appeal.

In terms of continuity, it seems that the second definition will not be continuos

Consider $p(\theta|y) = \mathcal{N}(0,1)$, $q_n(\theta) = \mathcal{N}(0,\frac{1}{n})$, such that $Q_n \overset{D}{\rightarrow} \delta_0$ Then we have

$$ D_{KL}(q_n||p) = -\mathbb{E}_{Q_n}[\ln p] - \mathbb{H}[Q_n] \\ = \frac{1}{2n} -\frac{1}{2} \ln \frac{1}{n} + C \overset{n \to \infty}{\rightarrow} \infty$$

Where the main point here seems to be that $\mathbb{H}$, when applied to a continuos distribution, i.e. the differential entropy, does not converge to the Shannon entropy of the limiting point mass, is that correct?

---

Edit:

Here I want to include my understanding of why the definition w.r.t. $P+Q$ yields $\infty$. Recall that for a dirac measure $Q = \delta_x$ we have

$$ \int\limits_{A} f dQ = f(x) \mathbb{I}[x \in A],$$ which we will repeatedly use. Also recall that the Radon-Nikodyn-density of a measure $M_1 << M_2$ is a nonnegative measurable function $\frac{dM_1}{dM_2}$ such that $M_1(A) = \int \limits_{A}\frac{dM_1}{dM_2} dM_2$ for all measurable sets $A$.

We have that

$$\frac{dP}{dP + dQ} = 1 - \mathbb{I}[\theta =x],$$

since $$\int\limits_{A} 1 - \mathbb{I}[\theta =x] dP + dQ = \\ \int\limits_{A} 1 - \mathbb{I}[\theta =x] dP + \int\limits_{A} 1 - \mathbb{I}[\theta =x] dQ \\ = \int\limits_{A} 1 - \mathbb{I}[\theta =x] dP + \mathbb{I}[x \in A](1 - \mathbb{I}[x=x]) \\ = \int\limits_{A} 1 - \mathbb{I}[\theta =x] dP + 0 = \int\limits_{A} 1 - \mathbb{I}[\theta =x] dP .$$

We have $$\frac{dQ}{dP + dQ} = \mathbb{I}[\theta =x],$$

since

$$ \int\limits_{A} \mathbb{I}[\theta =x] dP + dQ = \\ \int\limits_{A} \mathbb{I}[\theta =x] dP + \int\limits_{A} \mathbb{I}[\theta =x] dQ \\ = 0 + \mathbb{I}[x \in A] \mathbb{I}[x= x] =\mathbb{I}[x \in A] = \int\limits_{A} dQ$$

Now, the definition of the Kullback-Leibler-Divergence with respect to $P+Q$, that can be found in the linked post, is given by

$$D_{KL}(Q||P) = \int \frac{dQ}{dP + dQ} \frac{\frac{dQ}{dP + dQ}}{\frac{dP}{dP + dQ}} d P + Q$$

Plugging in the above obtained Radon-Nikodyn derivatives, we get

$$D_{KL}(Q||P) = \int \mathbb{I}[\theta = x] \ln \frac{\mathbb{I}[\theta = x]}{1- \mathbb{I}[\theta = x]} d P + Q \\ = \int \mathbb{I}[\theta = x] \ln \frac{\mathbb{I}[\theta = x]}{1- \mathbb{I}[\theta = x]} d P + \int \mathbb{I}[\theta = x] \ln \frac{\mathbb{I}[\theta = x]}{1- \mathbb{I}[\theta = x]} dQ \\ $$

Now recall that $P$ is absolutely continuos w.r.t. the Lebesque measure, and we have $$ \mathbb{I}[\theta = x] \ln \frac{\mathbb{I}[\theta = x]}{1- \mathbb{I}[\theta = x]} = 0 \ \text{Lebesque a.e.} $$

such that the first term above evaluates to 0.

For the second term we get

$$\int \mathbb{I}[\theta = x] \ln \frac{\mathbb{I}[\theta = x]}{1- \mathbb{I}[\theta = x]} dQ \\ = \mathbb{I}[x = x] \ln \frac{\mathbb{I}[x = x]}{1- \mathbb{I}[x = x]} \\ = \mathbb{I}[x = x] \ln \mathbb{I}[x = x] - \mathbb{I}[x = x] \ln( 1- \mathbb{I}[x = x]) \\ = 1 \ln 1 - 1 \ln 0 = \infty $$

Answer 1

Mine is not an answer, but just a try to make your correct procedure clearer.

Using Lebesgue theorem, you can consider the absolute continuous measure $P+Q$ as made of an absolute continuous measure $P$ and a singular $Q=\delta_x$.

To use the Radon-Nikodyn theorem, both the measures must be absolute continuous wrt $P+Q$ and $\sigma$-finite, which is true in your case.

Denoted $(\mathbb{X}_P, \mathcal{A}_P)$ and $(\mathbb{X}_Q, \mathcal{A}_Q)$ the disjoint measurable spaces with measures $P$ and $Q$, respectively, their union $(\mathbb{X}, \mathcal{A})$ is a measurable space with measure equal to $P+Q$.

As described here, given the densities $f$ and $g$ of $P$ and $Q$ wrt the measure $P+Q$ for the measurable space $(\mathbb{X},\mathcal{A})$ (these are given by Radon-Nikodyn theorem), one can write the KL:

$$ KL(P||Q)=\int_{\mathbb{X}} f\ln\frac{f}{g}\,d(P+Q) $$

For Radon-Nikodyn theorem, the density of $Q$ wrt $P+Q$ is the Dirac $\delta_x=\mathbb{I}[\theta=x]$, while the density of $P$ wrt $P+Q$ is $1-\mathbb{I}[\theta=x]$. Intuitively, this means that, in $\mathbb{X}$, $P$ is equal to itself in $\mathbb{X}_P$ as like as $Q$ in $\mathbb{X}_Q$.

These can be seen in the same way as you did (EDIT: fixed a mistake):

$$ \int_{\mathbb{X}}1-\mathbb{I}[\theta=x]\,d(P+Q)=\\ \int_{\mathbb{X}_P}1-\mathbb{I}[\theta=x]\,dP + \int_{\mathbb{X}_Q}1-\mathbb{I}[\theta=x]\,dQ=\\ \int_{\mathbb{X}_P}1-\mathbb{I}[\theta=x]\,dP+0=\\ \int_{\mathbb{X}_P}dP - \int_{\mathbb{X}_P}\mathbb{I}[\theta=x]\,dP=\\ \int_{\mathbb{X}_P}dP - 0 =\int_{\mathbb{X}}dP $$

given that $\int_{\mathbb{X}_Q} dP=0$, and

$$ \int_{\mathbb{X}}\mathbb{I}[\theta=x]\,d(P+Q)=\\ \int_{\mathbb{X}_P}\mathbb{I}[\theta=x]\,dP + \int_{\mathbb{X}_Q}\mathbb{I}[\theta=x]\,dQ=\\ 0+\int_{\mathbb{X}_Q}\mathbb{I}[\theta=x]\,dQ=\\ \int_{\mathbb{X}_Q}dQ=\int_{\mathbb{X}}dQ $$

given that $\int_{\mathbb{X}_P} dQ=0$.

The formulation of $KL(Q||P)$ in terms of transformed densities wrt to $P+Q$ is correct (using the densities of $P$ and $Q$ wrt $P+Q$):

$$ KL(Q||P)=\int_\mathbb{X} \ln\frac{dQ}{dP}\,dQ=\\ \int_\mathbb{X} \ln\left(\frac{dQ/d(P+Q)}{dP/d(P+Q)}\right)\,\frac{dQ}{d(P+Q)}d(P+Q) $$

Then, substituting the analytical expressions of $f$ and $g$ you get the result.