I am reading a tutorial on variational autoencoder. In any inference problem we make assumptions about the underlying process of data generation. In VAE's, The tutorial states that data is generated based on the equation

$$P(X) = \int P(X|z)p(z)dz$$

I would like to clarify how the underlying data is generated. Since $X$ the data is fixed, $P(X)$ is a constant.

Repeat multiple times (approximation)

  1. Sample $z \sim N(0,I)$
  2. Compute the likelihood of the data whereby $X = \{x_1,x_2,...,x_n\}$ and $P(X|z) = \prod_{i=1}^NN(x_i|f(z;\theta),\sigma^2*I)$

Take average over all sampled values of $z$ to compute data likelihood $P(X)$.

If my understanding is correct, It seems counterintuitive that in one sample of $z$, it is able to explain the likelihood over all training data in $P(X|Z)$.

Suppose $z=[0.1,0.5]$ for simplicity, and I use MNIST data of digit handwritings of 1-9. Then this particular $z$ value should be able to generate digits from 1-9 in the dataset ? How can this one value generate digits that are different from each other.

This is my understanding of the generative modelling. Correct me ifI am wrong.


2 Answers 2


The generative story describes how each image sample is generated. The story is as follows - (a) Sample z ~ N(z | 0, I); (b) Sample x ~ N(x | f_mu(z), f_sig(z))

For any generative story, going forward tells us about test time, and going backwards helps us learn the parameters (note that last step has x, our data).

Every model comes with its assumptions. The implicit assumption in VAE is : z-space is cleaved in such a way that different regions give different digits. Our goal is to figure out these regions. This is the "Encoder part".

We need to optimize the log-marginal-likelihood $logP(x)=log \Pi_i\int_z p(x_i | z)p(z)dz $. Let's say x_i is 7. Most of $p(x_i | z)$'s are going to be zero -- because most of the z's don't even give 7. We have "assumed" that only certain regions of z-space give 7. You are cluelessly computing $p(x_i | z)$ for z's that belong to 1,2,3etc clusters (because of $\int_z$ part). Encoder (parameterized by $\phi$) helps us to not be clueless. At training time, you encode x_i to z-space, and then decode it back to x-space. The math looks like this --

$$logP(x)=log \Pi_i\int_z \frac{p(x_i | z)q_{\phi}(z | x)p(z)}{q_{\phi}(z | x)}dz = \Sigma_i log\int_z \frac{p(x_i | z)q_{\phi}(z | x)p(z)}{q_{\phi}(z | x)}dz $$ Notice that we have an expectation wrt encoder q_phi inside log. So use Jensen inequality log(Expectation) >= Expectation(log) to get $$log P(x) \geq \Sigma_i E_{q_{\phi}}[log\frac{p(x_i, z)}{q_{\phi}(z|x)} ]$$ where RHS is the ELBO term. Effectively, (x_i)----$q_{\phi}$----(z)----$p_{\theta}$----(x_i)

Intuitively, the encoder q figures out how to take x to z in such a way that z is "rich" (cleaved), and the decoder p (parameterized by $\theta$) figures out to take z from this "rich" z-space to x.

At test time, you can't use encoder to take x to z. You don't even have x - your target is to generate x. So, you sample some random z from N(z | 0, I). From your question, let's say $z = [0.1, 0.5]$. The decoder $p_{\theta}$ "knows" that this z-space is rich, and the z you've sampled belongs to the 3 cluster (say). Hence, we generate the digit x = 3.

You sample some other z. The decoder figures that this belongs to the 1 cluster, and it generates x = 1.

Can just a 2-dimensional z-space be "rich" enough so that different regions correspond to each of the 10 digits? Maybe not. You trained your model assuming that it can.

  • $\begingroup$ hi @stbv thank you for your help. I think I sort of understand but I am still not sure. The encoder takes the data $x$ and maps it to the standard normal that we wish to sample from. The decoder is trained to output the original data $x$ from this standard normal distribution. Hence, the decoder essentially is trained to decode the original image from the standard normal $Z$ space ? $\endgroup$
    – calveeen
    Commented Dec 26, 2020 at 3:05
  • $\begingroup$ Yes, that is the training phase. Autoencoder family of models work like that. For example, in denoising autoencoder the input is (orig image + noise). Encoder takes this to z-space, and the decoder is trained to recover orig image (without noise). $\endgroup$
    – stbv
    Commented Dec 26, 2020 at 14:04

I think you're mixing up

  1. How to compute or approximately compute $P(X)$
  2. The "generative story"

Also, I might be wrong on this, but based on the last paragraph, it sounds like you're confusing $X$ with the entire dataset, whereas the notation in the article you linked uses $X$ to mean just one point in the dataset. $x_i$ are meant to be the individual components of vector $X$.

The two steps you described are one way to compute (1) -- density at some single data point $X$. As it turns out, this is too inefficient in practice, so people use the "variational lower bound", or "ELBO", to obtain a non stochastic, non exact, but guaranteed lower bound on $P(X)$.

The generative story goes like this:

  1. we sample some $z$ from a distribution $p(z)$, usually standard normal.
  2. compute $\mu = f(z;\theta)$
  3. we draw $X$ from $P(X|z)$ -- to be more precise, we draw from a normal distribution with the previously computed mean $\mu$.

Also regarding the last paragraph: you can abstractly think of $z$ as some "blueprint" for the generated data $X$. According to our generative story, a single $z$ can result in different $X$, because there's some randomness involved. And it is possible, although extremely unlikely, that a $z$ which usually generates the digit "3" might somehow generate the digit "8", because randomness is involved.

  • $\begingroup$ Thanks for the clarification regarding X being one datapoint. however, since $z$ is drawn from $N(0,1)$, I am still struggling to understand how samples from $N(0,1)$ can generate all the data(eg. MNIST digits). It would make more intuitive sense if the $z's$ used to generate different digits come from different distribution $\endgroup$
    – calveeen
    Commented Dec 21, 2020 at 10:40
  • $\begingroup$ Suppose you had a different $z$ distribution for each digit, and then suppose you had an even mixture of all those 10 distributions so you could draw a digit at random. Whatever this distribution is, as long as it's "reasonable", you can always convert it to a unit gaussian using some transform $f$. Plus, our neural network is sufficiently powerful that we can imagine it has such $f$ built-in, or learned. So we might as well use unit gaussian for simplicity. $\endgroup$
    – shimao
    Commented Dec 21, 2020 at 15:48
  • $\begingroup$ But the neural network does not compute the transform $f$ ? the $f$ is only used to map $z$ to $\mu$ $\endgroup$
    – calveeen
    Commented Dec 22, 2020 at 1:59
  • $\begingroup$ I overloaded the symbol $f$ by accident there, call it $g$ instead $\endgroup$
    – shimao
    Commented Dec 22, 2020 at 14:16
  • $\begingroup$ Might you know why for a single data point $X$, the marginal probability $P(X)$ needs to be an integral over the whole probability space $z$ ? (as mentioned in the tutorial) surely there would be some regions of the $z$ space that do not "represent" the single datapoint. This would lead to low likelihoods and for regions which "represent" the single data point, it would lead to high likelihoods. $\endgroup$
    – calveeen
    Commented Jan 6, 2021 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.