Return to Answer

Mine is not an answer, but just a try to make your correct procedure clearer.

Using Lebesgue theorem, you can consider the absolute continuous measure $P+Q$ as made of an absolute continuous measure $P$ and a singular $Q=\delta_x$.

To use the Radon-Nikodyn theorem, both the measures must be absolute continuous wrt $P+Q$ and $\sigma$-finite, which is true in your case.

Denoted $(\mathbb{X}_P, \mathcal{A}_P)$ and $(\mathbb{X}_Q, \mathcal{A}_Q)$ the disjoint measurable spaces with measures $P$ and $Q$, respectively, their union $(\mathbb{X}, \mathcal{A})$ is a measurable space with measure equal to $P+Q$.

As described here, given the densities $f$ and $g$ of $P$ and $Q$ wrt the measure $P+Q$ for the measurable space $(\mathbb{X},\mathcal{A})$ (these are given by Radon-Nikodyn theorem), one can write the KL:

$$ KL(P||Q)=\int_{\mathbb{X}} f\ln\frac{f}{g}\,d(P+Q) $$

For Radon-Nikodyn theorem, the density of $Q$ wrt $P+Q$ is the Dirac $\delta_x=\mathbb{I}[\theta=x]$, while the density of $P$ wrt $P+Q$ is $1-\mathbb{I}[\theta=x]$. Intuitively, this means that, in $\mathbb{X}$, $P$ is equal to itself in $\mathbb{X}_P$ as like as $Q$ in $\mathbb{X}_Q$.

These can be seen in the same way as you did (EDIT: fixed a mistake):

$$ \int_{\mathbb{X}}1-\mathbb{I}[\theta=x]\,d(P+Q)=\\ \int_{\mathbb{X}_P}1-\mathbb{I}[\theta=x]\,dP + \int_{\mathbb{X}_Q}1-\mathbb{I}[\theta=x]\,dQ=\\ \int_{\mathbb{X}_P}1-\mathbb{I}[\theta=x]\,dP+0=\\ \int_{\mathbb{X}_P}dP - \int_{\mathbb{X}_P}\mathbb{I}[\theta=x]\,dP=\\ \int_{\mathbb{X}_P}dP - 0 =\int_{\mathbb{X}}dP $$

and

$$ \int_{\mathbb{X}}\mathbb{I}[\theta=x]\,d(P+Q)=\\ \int_{\mathbb{X}_P}\mathbb{I}[\theta=x]\,dP + \int_{\mathbb{X}_Q}\mathbb{I}[\theta=x]\,dQ=\\ 0+\int_{\mathbb{X}_Q}\mathbb{I}[\theta=x]\,dQ=\\ \int_{\mathbb{X}_Q}dQ=\int_{\mathbb{X}}dQ $$

The formulation of $KL(Q||P)$ in terms of transformed densities wrt to $P+Q$ is correct (using the densities of $P$ and $Q$ wrt $P+Q$):

$$ KL(Q||P)=\int_\mathbb{X} \ln\frac{dQ}{dP}\,dQ=\\ \int_\mathbb{X} \ln\left(\frac{dQ/d(P+Q)}{dP/d(P+Q)}\right)\,\frac{dQ}{d(P+Q)}d(P+Q) $$

Then, substituting the analytical expressions of $f$ and $g$ you get the result.

Mine is not an answer, but just a try to make your correct procedure clearer.

Using Lebesgue theorem, you can consider the absolute continuous measure $P+Q$ as made of an absolute continuous measure $P$ and a singular $Q=\delta_x$.

To use the Radon-Nikodyn theorem, both the measures must be absolute continuous wrt $P+Q$ and $\sigma$-finite, which is true in your case.

Denoted $(\mathbb{X}_P, \mathcal{A}_P)$ and $(\mathbb{X}_Q, \mathcal{A}_Q)$ the disjoint measurable spaces with measures $P$ and $Q$, respectively, their union $(\mathbb{X}, \mathcal{A})$ is a measurable space with measure equal to $P+Q$.

As described here, given the densities $f$ and $g$ of $P$ and $Q$ wrt the measure $P+Q$ for the measurable space $(\mathbb{X},\mathcal{A})$ (these are given by Radon-Nikodyn theorem), one can write the KL:

$$ KL(P||Q)=\int_{\mathbb{X}} f\ln\frac{f}{g}\,d(P+Q) $$

For Radon-Nikodyn theorem, the density of $Q$ wrt $P+Q$ is the Dirac $\delta_x=\mathbb{I}[\theta=x]$, while the density of $P$ wrt $P+Q$ is $1-\mathbb{I}[\theta=x]$. Intuitively, this means that, in $\mathbb{X}$, $P$ is equal to itself in $\mathbb{X}_P$ as like as $Q$ in $\mathbb{X}_Q$.

These can be seen in the same way as you did (EDIT: fixed a mistake):

$$ \int_{\mathbb{X}}1-\mathbb{I}[\theta=x]\,d(P+Q)=\\ \int_{\mathbb{X}_P}1-\mathbb{I}[\theta=x]\,dP + \int_{\mathbb{X}_Q}1-\mathbb{I}[\theta=x]\,dQ=\\ \int_{\mathbb{X}_P}1-\mathbb{I}[\theta=x]\,dP+0=\\ \int_{\mathbb{X}_P}dP - \int_{\mathbb{X}_P}\mathbb{I}[\theta=x]\,dP=\\ \int_{\mathbb{X}_P}dP - 0 =\int_{\mathbb{X}}dP $$

given that $\int_{\mathbb{X}_Q} dP=0$, and

$$ \int_{\mathbb{X}}\mathbb{I}[\theta=x]\,d(P+Q)=\\ \int_{\mathbb{X}_P}\mathbb{I}[\theta=x]\,dP + \int_{\mathbb{X}_Q}\mathbb{I}[\theta=x]\,dQ=\\ 0+\int_{\mathbb{X}_Q}\mathbb{I}[\theta=x]\,dQ=\\ \int_{\mathbb{X}_Q}dQ=\int_{\mathbb{X}}dQ $$

given that $\int_{\mathbb{X}_P} dQ=0$.

The formulation of $KL(Q||P)$ in terms of transformed densities wrt to $P+Q$ is correct (using the densities of $P$ and $Q$ wrt $P+Q$):

$$ KL(Q||P)=\int_\mathbb{X} \ln\frac{dQ}{dP}\,dQ=\\ \int_\mathbb{X} \ln\left(\frac{dQ/d(P+Q)}{dP/d(P+Q)}\right)\,\frac{dQ}{d(P+Q)}d(P+Q) $$

Then, substituting the analytical expressions of $f$ and $g$ you get the result.

Mine is not an answer, but just a try to make your correct procedure clearer.

Using Lebesgue theorem, you can consider the absolute continuous measure $P+Q$ as made of an absolute continuous measure $P$ and a singular $Q=\delta_x$.

To use the Radon-Nikodyn theorem, both the measures must be absolute continuous wrt $P+Q$ and $\sigma$-finite, which is true in your case.

Denoted $(\mathbb{X}_P, \mathcal{A}_P)$ and $(\mathbb{X}_Q, \mathcal{A}_Q)$ the disjoint measurable spaces with measures $P$ and $Q$, respectively, their union $(\mathbb{X}, \mathcal{A})$ is a measurable space with measure equal to $P+Q$.

As described here, given the densities $f$ and $g$ of $P$ and $Q$ wrt the measure $P+Q$ for the measurable space $(\mathbb{X},\mathcal{A})$ (these are given by Radon-Nikodyn theorem), one can write the KL:

$$ KL(P||Q)=\int_{\mathbb{X}} f\ln\frac{f}{g}\,d(P+Q) $$

For Radon-Nikodyn theorem, the density of $Q$ wrt $P+Q$ is the Dirac $\delta_x=\mathbb{I}[\theta=x]$, while the density of $P$ wrt $P+Q$ is $1-\mathbb{I}[\theta=x]$. Intuitively, this means that, in $\mathbb{X}$, $P$ is equal to itself in $\mathbb{X}_P$ as like as $Q$ in $\mathbb{X}_Q$.

These can be seen in the same way as you did:

$$ \int_{\mathbb{X}}1-\mathbb{I}[\theta=x]\,d(P+Q)=\\ \int_{\mathbb{X}_P}1-\mathbb{I}[\theta=x]\,dP + \int_{\mathbb{X}_Q}1-\mathbb{I}[\theta=x]\,dQ=\\ \int_{\mathbb{X}_P}1-\mathbb{I}[\theta=x]\,dP+0=\\ \int_{\mathbb{X}}1-\mathbb{I}[\theta=x]\,dP $$

and

$$ \int_{\mathbb{X}}\mathbb{I}[\theta=x]\,d(P+Q)=\\ \int_{\mathbb{X}_P}\mathbb{I}[\theta=x]\,dP + \int_{\mathbb{X}_Q}\mathbb{I}[\theta=x]\,dQ=\\ 0+\int_{\mathbb{X}_Q}\mathbb{I}[\theta=x]\,dQ=\\ \int_{\mathbb{X}}\mathbb{I}[\theta=x]\,dQ $$

The formulation of $KL(Q||P)$ in terms of transformed densities wrt to $P+Q$ is correct (using the densities of $P$ and $Q$ wrt $P+Q$):

$$ KL(Q||P)=\int_\mathbb{X} \ln\frac{dQ}{dP}\,dQ=\\ \int_\mathbb{X} \ln\left(\frac{dQ/d(P+Q)}{dP/d(P+Q)}\right)\,\frac{dQ}{d(P+Q)}d(P+Q) $$

Then, substituting the analytical expressions of $f$ and $g$ you get the result.

Mine is not an answer, but just a try to make your correct procedure clearer.

Using Lebesgue theorem, you can consider the absolute continuous measure $P+Q$ as made of an absolute continuous measure $P$ and a singular $Q=\delta_x$.

To use the Radon-Nikodyn theorem, both the measures must be absolute continuous wrt $P+Q$ and $\sigma$-finite, which is true in your case.

Denoted $(\mathbb{X}_P, \mathcal{A}_P)$ and $(\mathbb{X}_Q, \mathcal{A}_Q)$ the disjoint measurable spaces with measures $P$ and $Q$, respectively, their union $(\mathbb{X}, \mathcal{A})$ is a measurable space with measure equal to $P+Q$.

As described here, given the densities $f$ and $g$ of $P$ and $Q$ wrt the measure $P+Q$ for the measurable space $(\mathbb{X},\mathcal{A})$ (these are given by Radon-Nikodyn theorem), one can write the KL:

$$ KL(P||Q)=\int_{\mathbb{X}} f\ln\frac{f}{g}\,d(P+Q) $$

For Radon-Nikodyn theorem, the density of $Q$ wrt $P+Q$ is the Dirac $\delta_x=\mathbb{I}[\theta=x]$, while the density of $P$ wrt $P+Q$ is $1-\mathbb{I}[\theta=x]$. Intuitively, this means that, in $\mathbb{X}$, $P$ is equal to itself in $\mathbb{X}_P$ as like as $Q$ in $\mathbb{X}_Q$.

These can be seen in the same way as you did (EDIT: fixed a mistake):

$$ \int_{\mathbb{X}}1-\mathbb{I}[\theta=x]\,d(P+Q)=\\ \int_{\mathbb{X}_P}1-\mathbb{I}[\theta=x]\,dP + \int_{\mathbb{X}_Q}1-\mathbb{I}[\theta=x]\,dQ=\\ \int_{\mathbb{X}_P}1-\mathbb{I}[\theta=x]\,dP+0=\\ \int_{\mathbb{X}_P}dP - \int_{\mathbb{X}_P}\mathbb{I}[\theta=x]\,dP=\\ \int_{\mathbb{X}_P}dP - 0 =\int_{\mathbb{X}}dP $$

and

$$ \int_{\mathbb{X}}\mathbb{I}[\theta=x]\,d(P+Q)=\\ \int_{\mathbb{X}_P}\mathbb{I}[\theta=x]\,dP + \int_{\mathbb{X}_Q}\mathbb{I}[\theta=x]\,dQ=\\ 0+\int_{\mathbb{X}_Q}\mathbb{I}[\theta=x]\,dQ=\\ \int_{\mathbb{X}_Q}dQ=\int_{\mathbb{X}}dQ $$

The formulation of $KL(Q||P)$ in terms of transformed densities wrt to $P+Q$ is correct (using the densities of $P$ and $Q$ wrt $P+Q$):

$$ KL(Q||P)=\int_\mathbb{X} \ln\frac{dQ}{dP}\,dQ=\\ \int_\mathbb{X} \ln\left(\frac{dQ/d(P+Q)}{dP/d(P+Q)}\right)\,\frac{dQ}{d(P+Q)}d(P+Q) $$

Then, substituting the analytical expressions of $f$ and $g$ you get the result.

I'm trying to understand your question. I hope I got it right.

As said in the comments and here, KL is well defined between a continuous distribution $f(\theta)$ and a discrete distribution $g=\delta(\theta)$, by using the definition

$$ \text{KL}(g||f)=\int_{\Theta}\delta(\theta)\ln\frac{\delta(\theta)}{f(\theta)}d\theta=\\ \int_\Theta\delta(\theta)\ln\delta(\theta)d\theta-\int_\Theta\delta(\theta)\ln f(\theta)d\theta=\\ 0-\ln f(0) $$

the first element is clearly the (minus) entropy of the $\delta(\theta)$ while the second term is equal $\mathbb{E}_{\delta(\theta)}[\ln(f(\theta))]$.

I don't see the problem of using a conditional distribution as you did. The KL in that case, is equal to $\mathbb{E}_{\delta(\theta)}[-\ln f(\theta|y)] = -\ln f(0|y)$

Mine is not an answer, but just a try to make your correct procedure clearer.

Using Lebesgue theorem, you can consider the absolute continuous measure $P+Q$ as made of an absolute continuous measure $P$ and a singular $Q=\delta_x$.

To use the Radon-Nikodyn theorem, both the measures must be absolute continuous wrt $P+Q$ and $\sigma$-finite, which is true in your case.

Denoted $(\mathbb{X}_P, \mathcal{A}_P)$ and $(\mathbb{X}_Q, \mathcal{A}_Q)$ the disjoint measurable spaces with measures $P$ and $Q$, respectively, their union $(\mathbb{X}, \mathcal{A})$ is a measurable space with measure equal to $P+Q$.

As described here, given the densities $f$ and $g$ of $P$ and $Q$ wrt the measure $P+Q$ for the measurable space $(\mathbb{X},\mathcal{A})$ (these are given by Radon-Nikodyn theorem), one can write the KL:

$$ KL(P||Q)=\int_{\mathbb{X}} f\ln\frac{f}{g}\,d(P+Q) $$

For Radon-Nikodyn theorem, the density of $Q$ wrt $P+Q$ is the Dirac $\delta_x=\mathbb{I}[\theta=x]$, while the density of $P$ wrt $P+Q$ is $1-\mathbb{I}[\theta=x]$. Intuitively, this means that, in $\mathbb{X}$, $P$ is equal to itself in $\mathbb{X}_P$ as like as $Q$ in $\mathbb{X}_Q$.

These can be seen in the same way as you did:

$$ \int_{\mathbb{X}}1-\mathbb{I}[\theta=x]\,d(P+Q)=\\ \int_{\mathbb{X}_P}1-\mathbb{I}[\theta=x]\,dP + \int_{\mathbb{X}_Q}1-\mathbb{I}[\theta=x]\,dQ=\\ \int_{\mathbb{X}_P}1-\mathbb{I}[\theta=x]\,dP+0=\\ \int_{\mathbb{X}}1-\mathbb{I}[\theta=x]\,dP $$

and

$$ \int_{\mathbb{X}}\mathbb{I}[\theta=x]\,d(P+Q)=\\ \int_{\mathbb{X}_P}\mathbb{I}[\theta=x]\,dP + \int_{\mathbb{X}_Q}\mathbb{I}[\theta=x]\,dQ=\\ 0+\int_{\mathbb{X}_Q}\mathbb{I}[\theta=x]\,dQ=\\ \int_{\mathbb{X}}\mathbb{I}[\theta=x]\,dQ $$

The formulation of $KL(Q||P)$ in terms of transformed densities wrt to $P+Q$ is correct (using the densities of $P$ and $Q$ wrt $P+Q$):

$$ KL(Q||P)=\int_\mathbb{X} \ln\frac{dQ}{dP}\,dQ=\\ \int_\mathbb{X} \ln\left(\frac{dQ/d(P+Q)}{dP/d(P+Q)}\right)\,\frac{dQ}{d(P+Q)}d(P+Q) $$

Then, substituting the analytical expressions of $f$ and $g$ you get the result.