Given two distributions, one a parameterized gaussian and the other a standard normal gaussian:

$q(x) \sim \mathcal{N}(\mu,\sigma)$

$p(x) \sim \mathcal{N}(0,I)$

We want to compute the KL Divergence $D_{KL}(q(x)||p(x))$. It is widely known that we can compute this in closed form solution such that the total KL divergence results in:

$= \sum_{i=0}^{D}(1 + \log(\sigma_i²) - \mu_i² - \sigma_i²)$

For a random vector with dimension $D$.

However, I tried to derive this from a different perspective and don't understand what I'm getting wrong... would really appreciate if someone could help me out here!

For a random variable $x \sim \mathcal{N}(\mu,\sigma)$, we can reparameterize it by drawing from a noise variable $\epsilon \sim \mathcal{N}(0,1)$ and setting $x = \mu + \sigma\epsilon$.

Next, the KL Divergence is given as:

$D_{KL}(q(x)||p(x)) = \int q(x)\: log\frac{q(x)}{p(x)} = \mathbb{E}_{q(x)}[q(x) - p(x)]$

The log of a Gaussian with $\mu,\sigma$ as:

$\log \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{1}{2}{\left(\frac{x-\mu}{\sigma}\right)}^{2}} = -\log{\sigma} - \frac{1}{2}\log(2\pi) - \frac{1}{2} {\left(\frac{x-\mu}{\sigma}\right)}^{2}$

And the log of a standard normal Gaussian with random variable $\epsilon$ as:

$\log \frac{1}{\sqrt{2\pi}} e^{-\frac{1}{2}{\epsilon}^{2}} = - \frac{1}{2}\log(2\pi) -\frac{1}{2}{\epsilon}^{2}$

So why can't we simply do:

$$ \begin{eqnarray} \mathbb{E}_{q(x)}[q(x) - p(x)] &=& \mathbb{E}_{q(x)}[\log{\sigma} - \frac{1}{2}\log(2\pi) - \frac{1}{2} {\left(\frac{x-\mu}{\sigma}\right)}^{2} - (- \frac{1}{2}\log(2\pi) -\frac{1}{2}{\epsilon}^{2})] \\ &=& \mathbb{E}_{q(x)}[\log(\sigma) - \frac{1}{2} {\left(\frac{x-\mu}{\sigma}\right)}^{2} -\frac{1}{2}{\epsilon}^{2}] \\ &=& \mathbb{E}_{p(\epsilon)}[\log(\sigma) - \frac{1}{2} {\left(\frac{\mu - \sigma\epsilon -\mu}{\sigma}\right)}^{2} +\frac{1}{2}{\epsilon}^{2}] \\ &=& \mathbb{E}_{p(\epsilon)}[\log(\sigma) - \frac{1}{2}{\epsilon}^{2} +\frac{1}{2}{\epsilon}^{2}] \\ &=& \mathbb{E}_{p(\epsilon)}[\log(\sigma)] \\ &=& \log(\sigma) \end{eqnarray} $$

Something is really missing here :( I thought that by we are allowed to plug-in the reparameterization of $x = \mu + \sigma\epsilon$ and thus change the distribution of expected value from $q(x)$ to $p(\epsilon)$.


1 Answer 1


Let's focus on the one-dimensional case. As you have shown, by definition, the KL-divergence $D_{\rm KL}(q(x) \vert p(x))$ is given by $$ \begin{aligned} D_{\rm KL}(q(x) \vert p(x)) &= \int dx \ q(x) \log\left(\frac{q(x)}{p(x)}\right)\\ &= {\rm E}_{q(x)}\left[\log q(x) - \log p(x)\right]. \end{aligned} $$ Following your steps, the KL-divergence $D_{\rm KL}(q(x) \vert p(x))$ for the two gaussian would be $$ \begin{aligned} D_{\rm KL}(q(x) \vert p(x)) &= {\rm E}_{q(x)}\left[\log q(x) - \log p(x)\right]\\ &={\rm E}_{q(x)}\left[-\log\sigma - \frac{1}{2}\log2\pi - \frac{1}{2}\left(\frac{x - \mu}{\sigma}\right)^2 - \left(- \frac{1}{2}\log2\pi - \frac{1}{2}x^2\right) \right]\\ &={\rm E}_{q(x)}\left[-\log\sigma - \frac{1}{2}\left(\frac{x - \mu}{\sigma}\right)^2 + \frac{1}{2}x^2\right], \end{aligned} $$ and you would like to do the calculation not in $x$ but in $\epsilon$, where $$ \epsilon = \left(\frac{x - \mu}{\sigma}\right). $$ The result would be $$ \begin{aligned} D_{\rm KL}(q(x) \vert p(x)) &= {\rm E}_{q(x)}\left[-\log\sigma - \frac{1}{2}\left(\frac{x - \mu}{\sigma}\right)^2 + \frac{1}{2}x^2\right]\\ &={\rm E}_{q(\epsilon)}\left[-\log\sigma - \frac{1}{2}\epsilon^2 + \frac{1}{2}(\mu + \sigma\epsilon)^2\right]\\ &=-\log\sigma - \frac{1}{2} + \frac{1}{2}\mu^2 + \frac{1}{2} \sigma^2, \end{aligned} $$ which is consistent with the result found here and here.

  • $\begingroup$ many thanks. i messed up with $\epsilon$ for the prior! $\endgroup$ Commented Apr 7, 2022 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.