1
$\begingroup$

In the "Auto-encoding Variation Bayes" Paper they state under "2.1 Problem Scenarios" that the VAE is a solution to:

"1. Efficient approximate ML or MAP estimation for the parameters $\theta$".

I am wondering if that makes sense. If one would just want to do an ML-Estimate for $\theta$ one would not need the recognition model/decoder $q_\phi$, right? One could just use the reparametrization trick on the model/encoder network $g_\theta$, which adds Gaussian noise on its last layer:

$$p_\theta(x) = E_{\epsilon}\left[~ \mathcal{N}\left(x ~|~ \mu(g^1_\theta(\epsilon)), \sigma(g^2_\theta(\epsilon)\right) ~\right]$$

where $\mu$ and $\sigma$ are outputs of a neural network. This would basically reduce to doing least squares on a generator network to fit the samples.

The VAE now basically also will do this but on top train the approximate posterior over $z$. Am I right in understanding, that this approximate posterior is the actual subject of interest, since otherwise just doing the above least squauares approach would be much simpler?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The expectation you wrote can't be computed efficiently and with an acceptable degree of accuracy -- about 3 paragraphs above where you quote, the authors describe this:

Intractability: the case where the integral of the marginal likelihood $p_\theta(x) = \int p_\theta(z) p_\theta(x|z) dz$ is intractable (so we cannot evaluate or differentiate the marginal likelihood) ... These intractabilities are quite common and appear in cases of moderately complicated likelihood functions pθ(x|z), e.g. a neural network with a nonlinear hidden layer

Improve this answer
$\endgroup$
3
  • $\begingroup$ Yes but the recognition model does not change this problem, does it? Now you have to evaluate and differentiate $$E_{q_\phi(z)}[\log p_\theta (x^{(i)} | z)]$$ later. This is done by the reparam. trick. Why not do the same here? I.e. choose $p_\theta(z)$ s.t. you can use the reparametrization trick and evaluate/differentiate the marginal likelihood $$p_\theta(x) = E_{p_\theta(z)}[\log p_\theta (x^{(i)} | z)].$$ $\endgroup$
    – Lochend
    Commented Jan 25, 2021 at 8:11
  • $\begingroup$ the recognition model doesn't make things tractable either, the use of a variational lower bound on the likelihood does. $\endgroup$
    – shimao
    Commented Jan 25, 2021 at 14:23
  • $\begingroup$ I see, we are now evaluating/differentiating something of the form $E_{q_\phi}[\log p(x | z)]$ instead of $E_{p_\theta}[p(x|z)]$. How does the logarithm make this tractable? $\endgroup$
    – Lochend
    Commented Jan 25, 2021 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.