Given a true normal distribution $g(x)$ with mean $\mu_G$ and variance $\sigma_G$, and a model $f(x)$, the KL divergence involves the expected log-likelihood $\mathbb{E}_G[log f(x|\theta]$. The likelihood function is given by $$log f(x|\mathbb{\theta})=-\frac{1}{2}log(2\pi\sigma^2)-\frac{(x-\mu)^2}{2\sigma^2}$$

I read, that $\mathbb{E}_G[log f(x)|\theta]$ equals $$-\frac{1}{2}log(2\pi\sigma^2)-\sigma_G^2+\frac{(\mu-\mu_G)^2}{\sigma^2}$$

but I do not understand how this result($\sigma_G^2+\frac{(\mu-\mu_G)^2}{\sigma^2}$) is obtained.


1 Answer 1


Hint: \begin{align} \int (x - \mu)^2 g(x) dx &= \int (x -\mu_G + \mu_G - \mu)^2 g(x) dx \\ &= \int(x - \mu_G)^2 g(x) dx + \int (\mu_G - \mu)^2 g(x) dx + 2\int (x - \mu_G)(\mu_G - \mu)g(x) dx \\ &=\sigma_G^2 + (\mu_G - \mu)^2 + 0 \end{align}


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.