VAE : How is likelihood $p(x|z)$ defined?

Question

Disclaimer : not a strong background in Bayesian statistics.

I gather from questions such as this one and this one that in the context of VAEs, we suppose that we know the (form of the ?) prior $p(z)$ and the likelihood $p(x|z)$. What we don't know is the posterior $p(z|x)$ and the marginal $p(x)$, the whole point of the VAE training process being to approximate them. However, I don't understand how we define the likelihood $p(x|z)$ and what form does it usually take.

Intuitively, I'd think that we would set it as some Gaussian for it be easily conjugable with the prior, to facilitate the computation of the posterior. Indeed, Wikipedia seems to say that $p(x|z)$ is usually defined as a Normal distribution :

• First, define a simple distribution $p(z)$ over a latent random variable $Z$. Usually a normal distribution or a uniform distribution suffices.
• Next, define a family of complicated functions $f_{\theta }$ (such as a deep neural network) parametrized by $\theta$ .
• Finally, define a way to convert any $f_{\theta }(z)$ into a simple distribution over the observable random variable $X$. For example, let $f_{\theta }(z)=(f_{1}(z),f_{2}(z))$ have two outputs, then we can define the corresponding distribution over $X$ to be the normal distribution $N(f_{1}(z),e^{f_{2}(z)})$.

But I thought that this was this "implictly parametrized distribution" "trick" was how we set $p(z|x)$ : As shown in the image below, it is $z$ which is sampled from a Normal distribution parametrized by $x$. In this image, $p(x|z)$ is defined as the function learned by a neural network, which is certainly not a "simple" function.

(image is from here)

Answer 1

The "whole point of the VAE training process" is to find an approximation to $p(z|x)$ [find the encoder] and to maximize $p(x|z)$ given the data [find the decoder].

In the original paper, they give 2 possibilities for the likelihood - either a Gaussian for continuous data, or Bernoulli for discrete data. E.g., Gaussian for images with a range of pixel value, and Bernoulli for 1-bit images (pixel is either black or white). [In practice you can also use Binary Cross Entropy (the negative log likelihood of Bernoulli distribution) also on grayscale images.]

As mentioned in appendix:

The "reparameterization trick" is used in the encoder, on $q(z|x)$, it's not related to the way you transform $z$ into $p(x|z)$. Both Gaussian and Bernoulli are simple distributions. Not functions. The way you get to their parameters can be very complex.

Answer 2

It depends on your output domain

• It can be a Multivariate Gaussian $$p(x|z) \sim \mathcal{N}(z, \mu_\phi(z),\Sigma_\phi(z))$$
• For images, you can also take a categorical distribution over the pixel values. $$p(x|z) \sim \text{Categorical}(x|\phi(z))$$ In practice, this translates to a neural network where you take the softmax over the pixel values.

You can then learn the parameterization $\phi$ of $p_\phi(x|z)$ by MLE, i.e. maximizing $$\mathbb{E}_{z \sim q_\theta(z)}[\log p_\phi(x|z)]$$

Indeed, Wikipedia seems to say that p(x|z) is usually defined as a Normal distribution

I think Wikipedia is talking about the parameterization of $q(z|x)$ in the passage you cited. $f_1(z)$ computes $\mu_\theta$ and $f_2(z)$ computes $\log \Sigma_\theta$. This is the distribution from where you sample $z \sim \mathcal{N}(z,\mu_\phi,\Sigma^2_\phi)$, which you then feed into the decoder.

You can find a detailed description of variational autoencoders here