# KL divergence between two univariate Gaussians

I need to determine the KL-divergence between two Gaussians. I am comparing my results to these, but I can't reproduce their result. My result is obviously wrong, because the KL is not 0 for KL(p, p).

I wonder where I am doing a mistake and ask if anyone can spot it.

Let $p(x) = N(\mu_1, \sigma_1)$ and $q(x) = N(\mu_2, \sigma_2)$. From Bishop's PRML I know that

$$KL(p, q) = - \int p(x) \log q(x) dx + \int p(x) \log p(x) dx$$

where integration is done over all real line, and that

$$\int p(x) \log p(x) dx = \frac{1}{2} (1 + \log 2 \pi \sigma_1^2),$$

so I restrict myself to $\int p(x) \log q(x) dx$, which I can write out as

$$-\int p(x) \log \frac{1}{(2 \pi \sigma_2^2)^{(1/2)}} e^{-\frac{(x-\mu_2)^2}{2 \sigma_2^2}} dx,$$

which I can separate into

$$\frac{1}{2} \log (2 \pi \sigma_2^2) - \int p(x) \log e^{-\frac{(x-\mu_2)^2}{2 \sigma_2^2}} dx.$$

Taking the log I get

$$\frac{1}{2} \log (2 \pi \sigma_2^2) - \int p(x) -\frac{(x-\mu_2)^2}{2 \sigma_2^2} dx,$$

where I separate the sums and get $\sigma_2^2$ out of the integral.

$$\frac{1}{2} \log (2 \pi \sigma_2^2) + \frac{\int p(x) x^2 dx - \int p(x) 2x\mu dx + \int p(x) \mu^2 dx}{2 \sigma_2^2}$$

Letting $\langle \rangle$ denote the expectation operator under $p$, I can rewrite this as

$$\frac{1}{2} \log (2 \pi \sigma_2^2) + \frac{\langle x^2 \rangle - 2 \langle x \rangle \mu_2 + \mu_2^2}{2 \sigma_2^2}.$$

We know that $var(x) = \langle x^2 \rangle - \langle x \rangle ^2$. Thus

$$\langle x^2 \rangle = \sigma_1^2 + \mu_1^2$$

and therefore

$$\frac{1}{2} \log (2 \pi \sigma^2) + \frac{\sigma_1^2 + \mu_1^2 - 2 \mu_1 \mu_2 + \mu_2^2}{2 \sigma_2^2},$$

which I can put as

$$\frac{1}{2} \log (2 \pi \sigma_2^2) + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2}.$$

Putting everything together, I get to

\begin{align*} KL(p, q) &= - \int p(x) \log q(x) dx + \int p(x) \log p(x) dx\\ &= \frac{1}{2} \log (2 \pi \sigma_2^2) + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2} - \frac{1}{2} (1 + \log 2 \pi \sigma_1^2)\\ &= \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2}. \end{align*} Which is wrong since it equals $1$ for two identical Gaussians.

Can anyone spot my error?

Update

Thanks to mpiktas for clearing things up. The correct answer is:

$KL(p, q) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2} - \frac{1}{2}$

-
please fix your latex, use $\log$ instead of $log$. –  mpiktas Feb 21 '11 at 10:51
sorry for posting the incorrect answer in the first place. I just looked at $x-\mu_1$ and immediately thought that the integral is zero. The point that it was squared completely missed my mind :) –  mpiktas Feb 21 '11 at 12:02
what about the multi variate case? –  user7001 Oct 23 '11 at 0:49

-