# Variational Autoencoder assumtions

I am currently reading the paper "Importance Weighted Autoencoders" and am having a hard time understanding something regarding the original Variational Autoencoder (VAE) as described here

In the first paragraph of the third subsection the author wrote this:

The VAE objective of Eqn. 3 heavily penalizes approximate posterior samples which fail to explain the observations. This places a strong constraint on the model, since the variational assumptions must be approximately satisfied in order to achieve a good lower bound. In particular, the posterior distribution must be approximately factorial and predictable with a feed-forward neural network. This VAE criterion may be too strict; a recognition network which places only a small fraction (e.g. 20%) of its samples in the region of high posterior probability region may still be sufficient for performing accurate inference.

The things in bold are the things that are not clear to me, where in the algorithm of the VAE do we constraint the posterior distribution to be approximatly factorial? and why is that constraint bad?
Additionally, how can does the new IWAE objective function relate to the last sentence of the paragraph? (the one with the 20%)

The VAE objective of Eqn. 3 heavily penalizes approximate posterior samples which fail to explain the observations.

Here's one possible interpretation. The variational lower bound on the log marginal likelihood (equation 3) can be rewritten as:

$$\log p(x) \ge E_{q(h \mid x)} \Big[ \log p(x \mid h) \Big] - D_{KL} \Big( q(h \mid x) \ \big\| \ p(h) \Big)$$

where $$x$$ contains observed variables, $$h$$ contains latent variables, $$p(x \mid h)$$ is the likelihood, $$p(h)$$ is the prior, and $$q(h \mid x)$$ is the variational posterior. The first term on the RHS is the expected log likelihood of the data, given latent variables drawn from the variational posterior. This term will be large if the variational posterior assigns high density to latent variable values that are likely to give rise to the data. That is, if the variational posterior and likelihood together "explain" the data well. Otherwise, this term—and therefore the variational lower bound—will be small.

In particular, the posterior distribution must be approximately factorial and predictable with a feed-forward neural network.

This could be rephrased as: A VAE seeks to approximate the true posterior with a factorized Gaussian distribution whose parameters are given by a neural net.

Maximizing the variational lower bound is also equivalent to minimizing $$D_{KL} \Big( q(h \mid x) \ \big\| \ p(h \mid x) \Big)$$, the KL divergence between the approximate and true posteriors. That is, we're searching for an approximate posterior that matches the true one as well as possible (as measured by KL divergence).

The original VAE paper (Kingma and Welling 2013) defines the variational posterior as a Gaussian distribution with isotropic covariance matrix (see equation 12 in appendix C.2):

$$q(h \mid x) = \mathcal{N} \Big( h \ \big| \ \mu(x), \sigma(x)^2 I \Big)$$

The conditional mean $$\mu(x)$$ and variance $$\sigma(x)^2$$ are outputs of a neural net that takes the observed variable $$x$$ as input. Since the covariance matrix $$\sigma(x)^2 I$$ is diagonal, the multivariate Gaussian distribution above factorizes as a product of univariate Gaussian distributions (one for each latent variable). In fact, it's even more restrictive than that, because the covariance matrix is isotropic, not just diagonal. So, all latent variables share the same conditional variance.