Why are we checking the difference between q(z|x), and p(z|x) in variational encoders?

Question

I'm trying to understand how VAEs work, because I didn't understand how cross entropy between $x$ (input fed into the encoder), and $p(x|z)$ (output of decoder) minus KL divergence between $p(z|x)$, and $p(z)$ result in the latent space being so continuous. Although, I still don't understand it, the first step is to understand why are we calculating the loss as $KL(q(z|x)||p(z|x))$, because as I read, this is the first "idea", but because of $p(x)$ is intractable, we use the fact that maximizing ELBO minimizes KL, and breaking down ELBO in an equation where we can calculate each piece.

The first question in my mind was why aren't we comparing $p(x|z)$, and $q(x|z)$. At the end of the day we care about how well $x$ was constructed from $z$. The encoder produces $q(z|x)$, but where does $p(z|x)$ come from, and what do we know about it? We're calculating $p(x|z)$ with the decoder, not $p(z|x)$. Or does it mean if we can transform $p(x|z)$ to $p(z|x)$, and the difference between $q(z|x)$, and $p(z|x)$ is small, then $p$ is predicting $x$ from $z$ accurately, as $p(x|z)$?

In this understanding, we're trying to build two networks mirroring each other. But I just don't understand seeing the current equations how it is performing well. We're comparing two distributions that maybe similar to each other, but in neural network perspective, can be wrong (poor reconstruction, or wrong place in latent space). So it shouldn't be enough to build networks that are exact opposite of each other. Our only pillar is $x$ fed into the encoder, and if it would mean $q(x|z)$, it would make sense, but as I've read so far, the input fed into the encoder is simply $x$. On the other side, if the input fed into encoder would be $q(x|z)$, it would prove that if the difference between $q(z|x)$, and $p(z|x)$ is small, it doesn't mean $p$ is predicting $x$ from $z$ accurately, as $p(x|z)$, at least early stages of the training.

So the question is does comparing $q(z|x)$ with $p(z|x)$ prove how effectively the encoder, and decoder mirroring each other? In my head they could mirror each other effectively by placing the latent variables not with the continuousity as they do, so how does the loss we're calculating ensures that? Also, regardless of the tractability of the breakdown, could comparing $q(x|z)$ with $p(x|z)$ measure the loss just as effective as comparing $q(z|x)$ with $p(z|x)$?

Are my thoughts, and imagination right, or I'm missing something important?

Sorry for asking questions that may be stupid, or obvious, but I don't have too much knowledge of probabilities, just built simple neural networks, and it is very hard to imagine in a programmer perspective.

Answer 1

The first thing to appreciate about VAEs is that they are not just some magical deep generative model but that they are a special case of the Auto-Encoding Variational Bayes algorithm for doing variational Bayesian inference in generative models.

What that means is we consider a setup with a dataset $\mathcal{D} =\{x_i\}_{i=1}^n$ where we assume the generative process for an instance $x_i$ is done in two steps:

1. First sample a latent variable $z_i$ from some prior $p(z)$
2. Second sample $x_i$ from the distribution $p(x|z)$

Note that we don't observe the values of $z_i$ in our dataset so we're interested in working out for each example in our data what's the posterior distribution over its corresponding latent variable given we've seen all the data - i.e. what is $p(z|x)$?

So in a sense then this is the main objective in VAEs - to uncover $p(z|x)$ and not exactly to model p(x), that is just a byproduct of the algorithm.

To answer your first question then, we are interested in the $D_{KL}[q(z|x)||(p(z|x)]$ because what we're doing is accepting that $p(z|x)$ is intractable and we won't be able to calculate it, but we're trying to approximate it with $q(z|x)$ (which here is using a neural network) and we want them to be as similar as possible, hence why we're minimising that divergence.

For your second question on why this should perform well you just need to take a look at the form of the ELBO given here: $$ \mathcal{L}(\theta,\phi) = \underbrace{\mathbb{E}_{q_\phi}[\log p_\theta(x|z)]}_{\text{Reconstruction Loss}} - \underbrace{D_{KL}[q_\phi(z|x)||p(z)]}_{\text{KL Regulariser}} $$ The first term is the expected likelihood of the data given the latent variables, it should make sense that is a sensible thing to maximise as this makes the data more likely under the model while the second term maintains the `auto-encoder' part and keeps the approximate distribution close to the prior.

For your final question I would just say that I don't think it's particularly useful to think of the two network as ``mirroring'' each other. Rather they both model a different probability distribution and it so happens that we can train them jointly and effectively through this auto encoding algorithm. Comparing $q(z|x)$ and $p(z|x)$ then is measuring how good the encoder is at approximating the true posterior distribution over the latent variables, and so doesn't really say anything about the decoder.

Also you repeatedly mention $q(x|z)$ but this is just not something that is part of the model since we kind of assume the structure of $p_\theta(x|z)$ and just learn that directly (it's the decoder) so we don't worry about some $q$ variational distribution.