2
$\begingroup$

Suppose that I have a a gaussian distribution $p(x;\mu,\sigma^{2})=N(x;\mu,\sigma^{2})$ and an improper distribution $p(x)\propto 1, x\in \mathbb{R}.$

I want to calculate the probabilistic distance between those two distributions, for that I use the Kullback–Leibler divergence.

$$KL(p(x;\mu,\sigma^{2}),p(x)) = \int p(x;\mu,\sigma^{2})\frac{p(x;\mu,\sigma^{2})}{p(x)}$$

Since, the $p(x)$ gives positive density to all the real line we have that $p(x;\mu,\sigma^{2})$ is absolutely continuous to the $p(x)$. So, we can move forward with the calculation of $KL$.

Then the $KL(p(x;\mu,\sigma^{2}),p(x))= log(\frac{1}{2\pi \sigma^{2}})+\sqrt{2\pi}|\sigma|^{3}$

Is this calculation correct?? I went through the link KL divergence between gaussian and uniform distribution and it made me wonder if my calculations are right.

$\endgroup$
2
  • $\begingroup$ You could view the improper distribution as a limit of Normal distributions with increasing variances. Apply the formula at stats.stackexchange.com/a/7449/919. $\endgroup$
    – whuber
    Sep 21, 2022 at 16:32
  • 1
    $\begingroup$ @whuber So, if I do that I'll take the limit $\sigma_{2} \rightarrow \infty$ then the $KL=\infty$? $\endgroup$
    – Fiodor1234
    Sep 21, 2022 at 16:52

1 Answer 1

0
$\begingroup$

You forgot a logarithm in your expression and you should use

$$KL(p(x;\mu,\sigma^{2}),p(x)) = \int p(x;\mu,\sigma^{2}) \log \left(\frac{p(x;\mu,\sigma^{2})}{p(x)}\right) dx$$


A problem with the improper prior $p(x) \propto 1$ is that the constant approaches zero and can not be eliminated. So the divergence will be infinite.


You can see the divergence as the the expectation of the log likelihood ratio. This likelihood will be infinite because the improper prior assigns zero probability to events that have non-zero probability given the normal distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.