20
$\begingroup$

I still don't understand why we force the distribution of the hidden representation of a Variational Autoencoder (VAE) to follow a multivariate normal distribution. Why this specific distribution and not another one ?

This is maybe linked with another question : Why is the weights distribution in a neural network following a Gaussian Distribution ? Is it just the application of the Central Limit theorem that tells you that many independent inputs will generate many independent errors, and the observed weights are the results of these multiple back-propagated signals... ?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ I thought it would be related to the central limit theory, and this paper may would be of help to you. $\endgroup$
    – Lerner Zhang
    Commented Apr 11, 2019 at 23:36
  • 1
    $\begingroup$ Many things in the world are not distributed normally but data scientists and computer scientists model them as Normal distributions anyways. Why? Because it is the most entropic (conservative) modelling decision that we can make for a random variable while still matching a particular expectation (average value) and variance (spread). ---CS109 $\endgroup$
    – Lerner Zhang
    Commented Apr 19, 2019 at 12:11

2 Answers 2

Reset to default
22
$\begingroup$

Normal distribution is not the only distribution used for latent variables in VAEs. There are also works using von Mises-Fisher distribution (Hypershperical VAEs [1]), and there are VAEs using Gaussian mixtures, which is useful for unsupervised [2] and semi-supervised [3] tasks.

Normal distribution has many nice properties, such as analytical evaluation of the KL divergence in the variational loss, and also we can use the reparametrization trick for efficient gradient computation (however, the original VAE paper [4] names many other distributions for which that works). Moreover, one of the apparent advantages of VAEs is that they allow generation of new samples by sampling in the latent space—which is quite easy if it follows Gaussian distribution. Finally, as @shimao remarked, it does not matter so much what distribution latent variables follow since using the non-linear decoder it can mimic arbitrarily complicated distribution of observations. It is simply convenient.

As for the second question, I agree with @shimao's answer.

[1]: Davidson, T.R., Falorsi, L., De Cao, N., Kipf, T. and Tomczak, J.M., 2018. Hyperspherical variational auto-encoders. arXiv preprint arXiv:1804.00891.

[2]: Dilokthanakul, N., Mediano, P.A., Garnelo, M., Lee, M.C., Salimbeni, H., Arulkumaran, K. and Shanahan, M., 2016. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv:1611.02648.

[3]: Kingma, D.P., Mohamed, S., Rezende, D.J. and Welling, M., 2014. Semi-supervised learning with deep generative models. In Advances in neural information processing systems (pp. 3581-3589).

[4]: Kingma, D.P. and Welling, M., 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your answer ! But would it work if we use a classic auto encoder (without any specific distribution for the hidden representation) and we sample vectors by approximating the distribution for the hidden representation... $\endgroup$
    – Tbertin
    Commented Apr 12, 2019 at 14:59
  • 2
    $\begingroup$ Perhaps, but the number of samples you need to get a good density estimate in the latent space grows exponentially with the dimensionality of the latent space. Also, density estimation itself is a tricky problem, so you may start with one problem and end up with two (which does not mean VAEs don't have their own difficulties, compared to standard AE)... $\endgroup$
    – Jan Kukacka
    Commented Apr 12, 2019 at 15:07
15
$\begingroup$

We use normal distribution because it is easily reparameterized. Also a sufficiently powerful decoder can map the normal distribution to any other distribution, so from a theoretical viewpoint, the exact choice is not important.

As for your second question, I would question your premise -- I am pretty sure weights are NOT normally distributed -- I recall seeing that resnet weights follow a more laplacian distribution. Anyway that is a pretty unrelated matter to the choice of prior in a VAE.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.