# Why do variational autoencoders not find the “best” latent variables?

From my understanding:

Variational autoencoders sample the latent variables $$y$$ using a proposal distribution $$q$$ of the observed variables $$x$$. The objective is that the decoder $$p$$ applied to $$y$$ defines a distribution whose value at $$x$$ is supposed to be maximized (reconstruction cost) plus a regularizer term.

Why not forget about $$q$$, and for every example $$x$$, iteratively update $$y$$ so as to maximize the objective? That is, setting $$y$$ to

$$\arg \max_y P(x | y)$$

rather than sampling it from $$q(y \mid x)$$.

Is $$q$$ an approximation motivated by inference speed?

Typically, we use the prior $$p(y) = \mathcal{N}(0, I)$$ and choose $$p(x|y) = \mathcal{N}(\mu, \sigma = f(y;\theta))$$ for some neural network $$f(\cdot; \theta)$$. The ELBO (evidence lower bound) is the objective of the VAE:

$$\log p(x) \geq E_{y \sim q(y|x)}[\log p(x|y)] - \mathcal{D}_{\text{KL}}(q(y|x)||p(y))$$

iteratively update $$y$$ so as to maximize the objective?

I'm not completely sure what you mean by "update $$y$$". There's no single $$y$$ value in our model to update, since it is a latent variable. We can only update our variational approximation of the posterior distribution of $$y$$ -- which we do by learning the parameters of the model of $$q$$.

Is q an approximation motivated by inference speed?

Yes, while training, we would like to sample $$y$$ from the true posterior distribution $$p(y|x) = \frac{1}{Z} p(y)p(x|y)$$ -- this would eliminate the gap between the ELBO and the true evidence $$\log p(x)$$ -- the inequality would become an equality. Unfortunately, sampling from this distribution is computationally intractable, so we resort to an approximation.

• I've updated my question to explain what I mean by iteratively update $y$. For many objectives, this $y$ is pretty easy to find. – Neil G Feb 17 at 7:12
• @NeilG ah I see what you mean. But in that case, you would be optimizing something which is no longer the ELBO, and thus no longer a lower bound on $\log p(x)$, which is our goal – shimao Feb 17 at 7:20
• That's what I'm asking. Ultimately, we want the best value of $y$, which is not the same as the one that optimizes ELBO. Why don't we just go ahead and find that. That's our real goal after all. – Neil G Feb 17 at 7:42
• @NeilG no, the ultimate goal of generative modeling is to learn the distribution of our data $x$ -- one way to operationalize this is to minimize the KL-divergence between our model $p(x)$ and the data distribution $D$, which is equivalent to maximizing $E_{x \sim D}[\log p(x)]$, which the ELBO gives us a lower bound on. we don't "actually" care about finding the value of $y$ which produces the best reconstruction -- reconstruction loss just happens to show up in the ELBO. – shimao Feb 17 at 9:09
• I agree with you up to the point where you say we're minimizing $E_{x\sim D} \log p(x \mid y)$. However, this $y$ can be defined using the distribution $q(y \mid x)$, or alternatively using the procedure I suggested. What I am asking for is why not use my procedure instead of q. What your answer is doing is restating the procedure that q uses. I understand that procedure. What I don't understand is why it's better than just using my procedure. – Neil G Feb 17 at 9:44