Explanation needed for VAE latence space cost term I'm studying VAEs (Variational Auto Encoders) from this link after failing to understand the official paper and stumbling upon the latence variables cost term:

The author seems to be trying to say that the probability distribution p(x) is equal to the max-likelihood estimation for parameters $\phi$ and $\theta$ plus the 'amount of information the approximated $q_\phi$ differs from the actual $p_\theta$).. I'm not sure what that means. Can somebody help me out with this?
 A: Before I go into details, have a look at section 2.2 of David Blei's VI review as well. Also note that the argument from above and why ELBO is good can also be deduced from using Jensen's inequality and some maths. The one below is just another way. 
For a second, forget about $\theta$ and $\phi$ and let's focus on rewriting the likelihood $log p(x)$ so that you can do max likelihood. Because we have some hidden/latent variables $z$ plus the parameters, the problem is not easily solved (intractability - $p(x) = \int p(z)p(x|z)dz$).  
Now, in their paper they refer to $q(z|x)$ as a probabilistic encoder, because this basically tells you how to generate the hidden vars $z$ from $x$. $q(z|x)$ is the approximation to the true posterior which by Bayes is $p(z|x) = \frac{p(x|z) p(z)}{p(x)}$
Now we're ready to write the KL divergence between the approximate encoder and the real one and show it's equivalent to your formula. 
\begin{equation}
\begin{aligned}
KL(q(z|x) || p(z|x)) &= \mathbb{E}_{q(z|x)}[log\frac{q(z|x)}{p(z|x)}]\geq 0 \\
&=\mathbb{E}_{q(z|x)}[log\frac{q(z|x) p(x)}{p(x|z) p(z)}]\geq 0 \\
&= \mathbb{E}_{q(z|x)}[log\frac{q(z|x)}{p(z)}] - \mathbb{E}_{q(z|x)}[log p(x|z)] + logp(x) \geq 0
\end{aligned}
\end{equation}
At the last step the expectation doesn't make sense since $p$ is independent of q. 
Rewriting this you get: 
\begin{equation}
\begin{aligned}
logp(x) &= - \mathbb{E}_{q(z|x)}[log\frac{q(z|x)}{p(z)}] + \mathbb{E}_{q(z|x)}[log p(x|z)] + KL(q(z|x) || p(z|x)) \\
&= ELBO + KL(q(z|x) || p(z|x))
\end{aligned}
\end{equation} which is your result. Note that you can prove the same thing using $q(z)$. So now the question is what you do with the KL at the end. Turns out you can't really solve that so you are satisfied with only minimizing ELBO. You can find a lot more about VI in one of the older papers from 1999. Introduction to variational
methods for graphical models
