2
$\begingroup$

I'm studying VAEs (Variational Auto Encoders) from this link after failing to understand the official paper and stumbling upon the latence variables cost term:

enter image description here

The author seems to be trying to say that the probability distribution p(x) is equal to the max-likelihood estimation for parameters $\phi$ and $\theta$ plus the 'amount of information the approximated $q_\phi$ differs from the actual $p_\theta$).. I'm not sure what that means. Can somebody help me out with this?

$\endgroup$
2
$\begingroup$

Before I go into details, have a look at section 2.2 of David Blei's VI review as well. Also note that the argument from above and why ELBO is good can also be deduced from using Jensen's inequality and some maths. The one below is just another way.

For a second, forget about $\theta$ and $\phi$ and let's focus on rewriting the likelihood $log p(x)$ so that you can do max likelihood. Because we have some hidden/latent variables $z$ plus the parameters, the problem is not easily solved (intractability - $p(x) = \int p(z)p(x|z)dz$).

Now, in their paper they refer to $q(z|x)$ as a probabilistic encoder, because this basically tells you how to generate the hidden vars $z$ from $x$. $q(z|x)$ is the approximation to the true posterior which by Bayes is $p(z|x) = \frac{p(x|z) p(z)}{p(x)}$

Now we're ready to write the KL divergence between the approximate encoder and the real one and show it's equivalent to your formula. \begin{equation} \begin{aligned} KL(q(z|x) || p(z|x)) &= \mathbb{E}_{q(z|x)}[log\frac{q(z|x)}{p(z|x)}]\geq 0 \\ &=\mathbb{E}_{q(z|x)}[log\frac{q(z|x) p(x)}{p(x|z) p(z)}]\geq 0 \\ &= \mathbb{E}_{q(z|x)}[log\frac{q(z|x)}{p(z)}] - \mathbb{E}_{q(z|x)}[log p(x|z)] + logp(x) \geq 0 \end{aligned} \end{equation} At the last step the expectation doesn't make sense since $p$ is independent of q. Rewriting this you get: \begin{equation} \begin{aligned} logp(x) &= - \mathbb{E}_{q(z|x)}[log\frac{q(z|x)}{p(z)}] + \mathbb{E}_{q(z|x)}[log p(x|z)] + KL(q(z|x) || p(z|x)) \\ &= ELBO + KL(q(z|x) || p(z|x)) \end{aligned} \end{equation} which is your result. Note that you can prove the same thing using $q(z)$. So now the question is what you do with the KL at the end. Turns out you can't really solve that so you are satisfied with only minimizing ELBO. You can find a lot more about VI in one of the older papers from 1999. Introduction to variational methods for graphical models

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.