Kullback–Leibler divergence between normal distribution and improper distribution

Question

Suppose that I have a a gaussian distribution $p(x;\mu,\sigma^{2})=N(x;\mu,\sigma^{2})$ and an improper distribution $p(x)\propto 1, x\in \mathbb{R}.$

I want to calculate the probabilistic distance between those two distributions, for that I use the Kullback–Leibler divergence.

$$KL(p(x;\mu,\sigma^{2}),p(x)) = \int p(x;\mu,\sigma^{2})\frac{p(x;\mu,\sigma^{2})}{p(x)}$$

Since, the $p(x)$ gives positive density to all the real line we have that $p(x;\mu,\sigma^{2})$ is absolutely continuous to the $p(x)$. So, we can move forward with the calculation of $KL$.

Then the $KL(p(x;\mu,\sigma^{2}),p(x))= log(\frac{1}{2\pi \sigma^{2}})+\sqrt{2\pi}|\sigma|^{3}$

Is this calculation correct?? I went through the link KL divergence between gaussian and uniform distribution and it made me wonder if my calculations are right.

Answer 1

You forgot a logarithm in your expression and you should use

$$KL(p(x;\mu,\sigma^{2}),p(x)) = \int p(x;\mu,\sigma^{2}) \log \left(\frac{p(x;\mu,\sigma^{2})}{p(x)}\right) dx$$

---

A problem with the improper prior $p(x) \propto 1$ is that the constant approaches zero and can not be eliminated. So the divergence will be infinite.

---

You can see the divergence as the the expectation of the log likelihood ratio. This likelihood will be infinite because the improper prior assigns zero probability to events that have non-zero probability given the normal distribution.