Why is Bayesian inference necessary for generative models?

Question

I'm trying to understand and summarise the equations for Bayesian inference and generative models in order to motivate the use of deep generative models (such as VAE).

A generative model is given by the joint probability distribution

$$p(x,z) = p(x|z)p(z)$$

with the latent variable $z$ being a unit gaussian process. The likelihood is given by

$$ p(x|z) = \dfrac{p(z|x)p(x)}{p(z)} $$

We know that the marginal likelihood (evidence) is intractable due to the integral given by

$$ p(x) = \int p(x|z)p(z)dz $$

Re-arranging everything yields $$p(x|z) = \dfrac{p(z|x) \int p(x|z)p(z)dz }{p(z)}$$

So the problem we are facing is that we can't find the evidence. However, in literature everyone is concerned about finding the true posterior $p(z|x)$ by re-arranging the second equation. Then the posterior becomes intractable due to the intractable evidence. Apparently finding the likelihood is not the problem. But how does that relate to the generative model in the first equation since that contains the likelihood?

Basically, I'm confused about how these equations exactly relate to one another, where the problem really lies and how these deep generative models (VAE in particular) solve them. Why do we perform inference to create a generative model when a generative model is defined by the likelihood and prior?

Note: I do know very well how a VAE works, I am just confused by the fundamental theory behind what problem they solve.

Answer 1

I'll give a perspective on this question in your post:

Why do we perform inference to create a generative model when a generative model is defined by the likelihood and prior?

The generative model for particular observation $x$ $$ p(x) = \int p(x|z) p(z) dz \approx \sum_k p(x|z_k) p(z_k) $$ which can be considered as a mixture model over all possible values of $z$. In training however it would be inefficient to consider all possible values of $z$ because most of them correspond to $p(z)$ that is near zero. Instead, a VAE simultaneously trains an encoder (approximate posterior) that maps an observed datup $x$ to a distribution $z$ that could have produced that $x$.

In implementations almost always a single sample from the encoder distribution is taken, so it seems weird to talk about a mixture, but in a few papers the use of multiple samples is discussed (and I think at least one argues that the quality improves slightly when this is done). Regardless, I think the original thinking for VAEs considered the implementation (even with one sample) as an approximation of the continuous generative model.