Following is from the original paper of concept of VAE(variational autoencoder) by Kingma,Welling 2014

B. Solution of $D_{KL}(p_\phi(z)||q_\theta(z))$ of Gaussian case

The variational lower bound (the objective to be maximized) contains a KL term that can often be integrated analytically. Here we give the solution when both the prior $q_{\theta}(z) = N (0, I)$ and the posterior approximation $p_\phi(z|x^{ (i)})$ are Gaussian. Let $J$ be the dimensionality of $z$. Let $\mu$ and $\sigma$ denote the variational mean and standard deviation evaluated at datapoint $i$, and let $\mu_j$ and $\sigma_j$ simply denote the $j$-th element of these vectors. Then: $$\int p_\theta(z) \log q(z) dz\\ = \int N (z; \mu,\sigma^2 ) \log N (z; 0, I) dz\\ = − {J\over 2} \log(2\pi) − {1\over 2} \sum_{j=1}^{J} (\mu_j{^2} + \sigma_j^2 )$$

At the equation above can't understand how the second equality calculated. Any hint to understand those eqaulity?


In this case, I find it easier to start from a simple case and then build up in complexity. The simplest case is to consider $J=1$.

$$ \begin{align} -\int_{-\infty}^{\infty} p(z) \log(q(z)) dz &= \frac{1}{2}\log(2\pi\sigma_2^2) - \int p(z) \left(-\frac{\left(z - \mu_2\right)^2}{2 \sigma_2^2}\right)dz \\ &= \frac{1}{2}\log(2\pi\sigma_2^2) + \frac{\mathbb{E}_{z\sim p}[z^2] - 2 \mathbb{E}_{z\sim p}[z]\mu_2 +\mu_2^2} {2\sigma_2^2} \\ &= \frac{1}{2}\log(2\pi\sigma_2^2) + \frac{\sigma_1^2 + \mu_1^2-2\mu_1\mu_2+\mu_2^2}{2\sigma_2^2} \\ &= \frac{1}{2}\log(2\pi\sigma_2^2) + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2\sigma_2^2}\\ \int_{-\infty}^{\infty} p(z) \log(q(z)) dz = V &= -\frac{1}{2}\log(2\pi\sigma_2^2)-\frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2\sigma_2^2} \end{align} $$ The key is recognizing that we can expand the quadratic in the first line; this gives us a sum of several integrals. Then we apply the law of the unconscious statistician, and we use the fact that $\text{Var}(z)=\mathbb{E}[z^2]-\mathbb{E}[z]^2$. The rest is just rearranging.

In this special case, we know that $q$ is a standard normal, so $$ \begin{align} -\frac{1}{2}\log(2\pi\sigma_2^2)-\frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2\sigma_2^2} &= -\frac{1}{2} \log(2\pi) -\frac{1}{2}\left(\sigma_1^2 + \mu_1^2\right) \end{align} $$

How can we generalize this to $J>1$? Consider the case of a diagonal covariance matrix. In this case, the $z_j$ are independent. So the solution arises from the sum of $V_j$. If you're not convinced, then you'll need to crank through the matrix arithmetic, using multivariate normal $p$ and $q$. It's not particularly hard, it's just tedious. Here's some threads to get started:


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.