0
$\begingroup$

I've read the original paper and many blogs and it only confused me more due to various conflicting notations being used. So I'm looking for a canonical answer here hopefully.

All the following statements are based on a standard variational autoencoder.

Confusions:

  1. I've seen the word prior and posterior thrown around a lot. I've seen posterior refer to the distribution $p(x)$. But it is actually referring to the reconstructed input $x$, not the input that is fed to the encoder. Similarly, the prior is $p(z)$. From my understanding of priors and posteriors in a general sense, the prior should be the original input and the posterior is after new information has been incorporated. So let $x, z, \tilde x$ refer to the input, latent variable, and reconstruction. Which of these variables would you say is the prior, posterior, etc?

  2. The "posterior" is also referred to as the "evidence" since it is the evidence of the "prior." Can the "prior" be considered the evidence with regards to the input?

  3. Why is it called evidence "lower bound?" I get the loss function and how it is used, e.g. maximizing the log liklihood or minimzing the negative log liklihood, etc. But why exactly are we calling it "evidence lower bound" for the loss function function of a VAE?

  4. The decoder seems just as important as the encoder. But why are all the diagrams of VAE only concerned with $p(z | x)$ and not so much on $p( \tilde x | z)$? Maybe I just didn't catch the discussion surrounding this.

$\endgroup$
1
$\begingroup$
  1. The prior and posterior distributions are distributions over your parameters/latent variables, not your data. So in this case, the prior would be $p(z)$, and the posterior is $p(z|x)$.

  2. I have never seen the posterior referred to as the evidence. Sometimes, you may see the marginal likelihood $p(x)$ referred to as the model evidence.

  3. As mentioned above, denote the log model evidence as the log of the marginal likelihood of the data:

$$\log p(x) = \log \int_z p(x, z)dz.$$

We can do some simple algebra and apply Jensen's inequality:

\begin{align*} \log p(x) &= \log \int_z p(x, z)dz \\ &= \log \int_z p(x, z)\frac{q(z)}{q(z)}dz \\ &= \log \mathbb{E}_q\left[\frac{p(x, z)}{q(z)}\right] \\ &\geq \mathbb{E}_q\left[\log \frac{p(x, z)}{q(z)}\right] \\ &= \mathbb{E}_q[\log p(x, z)] - \mathbb{E}_q[\log q(z)]. \end{align*}

This last term is the ELBO. As you can see, we have $\log p(x) \geq \text{ELBO}$, meaning that the ELBO lower-bounds the model evidence.

  1. Sometimes, the purpose of a variational autoencoder is to compute a low dimensional approximation to our data that still achieves good reconstruction accuracy. If this is your goal, then $p(z|x)$ is the important part, since you don't care about reconstructing the input, you just care about a good approximation. However, variational autoencoders can also be used as generative models to simulate new data that was not part of the original dataset. In this case, you would be more interested in the decoder $p(\tilde{x} | z)$.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.