For variational autoencoder, how can we fix prior and likelihood distributions?

Question

Before starting to describe my question, sorry for my poor English if you can't understand my descriptions.

To my knowledge, generally, the Gaussian distribution is choosed as prior in VAE. Also, the likelihood distributions are fixed as specific distributions like Gaussian, Bernoulli, Poisson, ... to approximate ELBO(evidence lowerbound) easier.

And, I understood that normal distributions have some nice properties for constructing neural networks and sampling.

However, I'm confused how can we fix prior and likelihood distributions. What if the true distribution, p(x|θ), is too complex to approximate after fixing prior and likelihood? Is it enough to fix prior and likelihood to predict arbitray distribution p(x|θ)?

I tried to find proper references for the question, but I cannot find.

Thanks for reading my question.

Answer 1

Arbitrarily defining the likelihood $\; p(x|\theta) \;$ and the prior $\; p(\theta) \;$ is not a specific trait of the variational autoencoder. Rather, it is how Bayesian models in general are built. The likelihood is chosen to be sensible based on our knowledge of the problem, and the prior is typically set to a very generic distribution such as an isotropic Gaussian.

You are right though in that there is no way to know whether the likelihood chosen perfectly emulates the system in real life. But this problem is not specific to Bayesian modelling. In statistics, we usually value models based on how useful they are in practice, rather than on theoretical perfection. As Box said: "All models are wrong, but some are useful". For example, you have the same problem in frequentist machine learning: how do you know that the architecture of a neural network and the optimized parameters are the absolute best you can find?

What we can do, however, is to compare two models and choose one based on the data available. Such a thing is called a Bayes factor. Consider two candidate models $\; \mathcal{M}_1, \mathcal{M}_2 \;$ defined to model data $\; \mathcal{D} = \{ x_i \}_{i=1}^n \;$. A Bayes factor computes the evidence $\; p( \mathcal{D}| \mathcal{M}) = \int p( \mathcal{D}| \theta, \mathcal{M}) \; p(\theta | \mathcal{M}) \;\textrm{d}\theta \;$ for each model, and chooses the model with the highest value. It is called Bayes factor because we can express the decision with the following ratio:

$$ \frac{ p( \mathcal{D} | \mathcal{M}_1)}{ p( \mathcal{D} | \mathcal{M}_2)} $$

If the ratio is larger than 1, we choose $\;\mathcal{M}_1$, and if it is smaller than 1, we choose $\;\mathcal{M}_2$.