Generative story for a variational auto encoder

Question

I am reading a tutorial on variational autoencoder. In any inference problem we make assumptions about the underlying process of data generation. In VAE's, The tutorial states that data is generated based on the equation

$$P(X) = \int P(X|z)p(z)dz$$

I would like to clarify how the underlying data is generated. Since $X$ the data is fixed, $P(X)$ is a constant.

Repeat multiple times (approximation)

1. Sample $z \sim N(0,I)$
2. Compute the likelihood of the data whereby $X = \{x_1,x_2,...,x_n\}$ and $P(X|z) = \prod_{i=1}^NN(x_i|f(z;\theta),\sigma^2*I)$

Take average over all sampled values of $z$ to compute data likelihood $P(X)$.

If my understanding is correct, It seems counterintuitive that in one sample of $z$, it is able to explain the likelihood over all training data in $P(X|Z)$.

Suppose $z=[0.1,0.5]$ for simplicity, and I use MNIST data of digit handwritings of 1-9. Then this particular $z$ value should be able to generate digits from 1-9 in the dataset ? How can this one value generate digits that are different from each other.

This is my understanding of the generative modelling. Correct me ifI am wrong.

Answer 1

The generative story describes how each image sample is generated. The story is as follows - (a) Sample z ~ N(z | 0, I); (b) Sample x ~ N(x | f_mu(z), f_sig(z))

For any generative story, going forward tells us about test time, and going backwards helps us learn the parameters (note that last step has x, our data).

Every model comes with its assumptions. The implicit assumption in VAE is : z-space is cleaved in such a way that different regions give different digits. Our goal is to figure out these regions. This is the "Encoder part".

We need to optimize the log-marginal-likelihood $logP(x)=log \Pi_i\int_z p(x_i | z)p(z)dz $. Let's say x_i is 7. Most of $p(x_i | z)$'s are going to be zero -- because most of the z's don't even give 7. We have "assumed" that only certain regions of z-space give 7. You are cluelessly computing $p(x_i | z)$ for z's that belong to 1,2,3etc clusters (because of $\int_z$ part). Encoder (parameterized by $\phi$) helps us to not be clueless. At training time, you encode x_i to z-space, and then decode it back to x-space. The math looks like this --

$$logP(x)=log \Pi_i\int_z \frac{p(x_i | z)q_{\phi}(z | x)p(z)}{q_{\phi}(z | x)}dz = \Sigma_i log\int_z \frac{p(x_i | z)q_{\phi}(z | x)p(z)}{q_{\phi}(z | x)}dz $$ Notice that we have an expectation wrt encoder q_phi inside log. So use Jensen inequality log(Expectation) >= Expectation(log) to get $$log P(x) \geq \Sigma_i E_{q_{\phi}}[log\frac{p(x_i, z)}{q_{\phi}(z|x)} ]$$ where RHS is the ELBO term. Effectively, (x_i)----$q_{\phi}$----(z)----$p_{\theta}$----(x_i)

Intuitively, the encoder q figures out how to take x to z in such a way that z is "rich" (cleaved), and the decoder p (parameterized by $\theta$) figures out to take z from this "rich" z-space to x.

At test time, you can't use encoder to take x to z. You don't even have x - your target is to generate x. So, you sample some random z from N(z | 0, I). From your question, let's say $z = [0.1, 0.5]$. The decoder $p_{\theta}$ "knows" that this z-space is rich, and the z you've sampled belongs to the 3 cluster (say). Hence, we generate the digit x = 3.

You sample some other z. The decoder figures that this belongs to the 1 cluster, and it generates x = 1.

Can just a 2-dimensional z-space be "rich" enough so that different regions correspond to each of the 10 digits? Maybe not. You trained your model assuming that it can.

Answer 2

I think you're mixing up

1. How to compute or approximately compute $P(X)$
2. The "generative story"

Also, I might be wrong on this, but based on the last paragraph, it sounds like you're confusing $X$ with the entire dataset, whereas the notation in the article you linked uses $X$ to mean just one point in the dataset. $x_i$ are meant to be the individual components of vector $X$.

The two steps you described are one way to compute (1) -- density at some single data point $X$. As it turns out, this is too inefficient in practice, so people use the "variational lower bound", or "ELBO", to obtain a non stochastic, non exact, but guaranteed lower bound on $P(X)$.

The generative story goes like this:

1. we sample some $z$ from a distribution $p(z)$, usually standard normal.
2. compute $\mu = f(z;\theta)$
3. we draw $X$ from $P(X|z)$ -- to be more precise, we draw from a normal distribution with the previously computed mean $\mu$.

Also regarding the last paragraph: you can abstractly think of $z$ as some "blueprint" for the generated data $X$. According to our generative story, a single $z$ can result in different $X$, because there's some randomness involved. And it is possible, although extremely unlikely, that a $z$ which usually generates the digit "3" might somehow generate the digit "8", because randomness is involved.