From my understanding:

Variational autoencoders sample the latent variables $y$ using a proposal distribution $q$ of the observed variables $x$. The objective is that the decoder $p$ applied to $y$ defines a distribution whose value at $x$ is supposed to be maximized (reconstruction cost) plus a regularizer term.

Why not forget about $q$, and for every example $x$, iteratively update $y$ so as to maximize the objective? That is, setting $y$ to

$$\arg \max_y P(x | y)$$

rather than sampling it from $q(y \mid x)$.

Is $q$ an approximation motivated by inference speed?


Typically, we use the prior $p(y) = \mathcal{N}(0, I)$ and choose $p(x|y) = \mathcal{N}(\mu, \sigma = f(y;\theta))$ for some neural network $f(\cdot; \theta)$. The ELBO (evidence lower bound) is the objective of the VAE:

$$\log p(x) \geq E_{y \sim q(y|x)}[\log p(x|y)] - \mathcal{D}_{\text{KL}}(q(y|x)||p(y))$$

iteratively update $y$ so as to maximize the objective?

I'm not completely sure what you mean by "update $y$". There's no single $y$ value in our model to update, since it is a latent variable. We can only update our variational approximation of the posterior distribution of $y$ -- which we do by learning the parameters of the model of $q$.

Is q an approximation motivated by inference speed?

Yes, while training, we would like to sample $y$ from the true posterior distribution $p(y|x) = \frac{1}{Z} p(y)p(x|y)$ -- this would eliminate the gap between the ELBO and the true evidence $\log p(x)$ -- the inequality would become an equality. Unfortunately, sampling from this distribution is computationally intractable, so we resort to an approximation.

  • $\begingroup$ I've updated my question to explain what I mean by iteratively update $y$. For many objectives, this $y$ is pretty easy to find. $\endgroup$
    – Neil G
    Feb 17 '20 at 7:12
  • $\begingroup$ @NeilG ah I see what you mean. But in that case, you would be optimizing something which is no longer the ELBO, and thus no longer a lower bound on $\log p(x)$, which is our goal $\endgroup$
    – shimao
    Feb 17 '20 at 7:20
  • $\begingroup$ That's what I'm asking. Ultimately, we want the best value of $y$, which is not the same as the one that optimizes ELBO. Why don't we just go ahead and find that. That's our real goal after all. $\endgroup$
    – Neil G
    Feb 17 '20 at 7:42
  • $\begingroup$ @NeilG no, the ultimate goal of generative modeling is to learn the distribution of our data $x$ -- one way to operationalize this is to minimize the KL-divergence between our model $p(x)$ and the data distribution $D$, which is equivalent to maximizing $E_{x \sim D}[\log p(x)]$, which the ELBO gives us a lower bound on. we don't "actually" care about finding the value of $y$ which produces the best reconstruction -- reconstruction loss just happens to show up in the ELBO. $\endgroup$
    – shimao
    Feb 17 '20 at 9:09
  • $\begingroup$ I agree with you up to the point where you say we're minimizing $E_{x\sim D} \log p(x \mid y)$. However, this $y$ can be defined using the distribution $q(y \mid x)$, or alternatively using the procedure I suggested. What I am asking for is why not use my procedure instead of q. What your answer is doing is restating the procedure that q uses. I understand that procedure. What I don't understand is why it's better than just using my procedure. $\endgroup$
    – Neil G
    Feb 17 '20 at 9:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.