Autoencoders want to minimize distortion of encoding-decoding process, preferably alongside evaluation by discriminant.

Generative autoencoders additionally would like latent variable from a chosen probability distribution, usually (multivariate) Gaussian.

While optimization to get sample looking like from Gaussian might sound simple, it isn't - it would be great to understand what do we really want here?

Many approaches were recently proposed, for example:

  • The original Variational AutoEncoder (VAE) directly tested only KL for separate points – not ensuring their uniform coverage.
  • A year ago there was proposed using Wasserstein metric (WAE, SWAE) to optimize distance between the entire sample and multivariate Gaussian. However, due to difficulty of calculation, it represents the distribution with a random sample, then still requires approximation.
  • Half a year ago there was finally proposed a non-random analytic formula: by calculating L2 distance between 1D projections of Gaussian-smoothened sample, and averaging over all projection directions (CWAE). But still this is only guessing that we will get the desired distribution – tested usually only by verifying two moments (Marida test), still leaving huge freedom for continuous distributions.
  • We can also directly optimize empirical distribution function to agree with CDF of Gaussian distribution, especially for radii $\|x_i\|$ and distances $\|x_i -x_j\|$.

As there are many ways to approach this problem, it would be great to specify it - understand what do we want to achieve here?

What do we really mean by "latent variables from Gaussian distribution"? How important it is?


1 Answer 1


I can think of at least two reasons why it seems desirable for the latents to come from a Gaussian distribution:

  1. Sampling

It's easy to sample random numbers from a Gaussian, so then drawing samples from our generative distribution, $p(x)$, is easy because we first sample $ z \sim \mathcal N(0,\mathrm I)$, then sample each variable $x_i \sim p(x_i|z)$ (since our generative model usually assumes conditionally independent variables). For this purpose, there seem to be many other distributions that would suffice.

  1. Per-sample likelihood calculation

For VAE, we can analytically calculate a lower bound on the likelihood of generating a single sample point, x, using the ELBO. To get this to work, the KL divergence of the encoder has to "match" the prior in form. So if our encoder, $q(z|x)$ adds Gaussian noise, then we can make the prior, $p(z)$, Gaussian and get a tractable expression. If we change $p(z)$, we'll have to change the form of q(z|x) as well. Even then, there are many distributions where an analytic form for $D_{KL} (q(z|x)||p(z))$ would not be available.

In practice, I think your intuition is correct that there are many degrees of freedom in the optimization and most autoencoders don't actually achieve this desiderata (i.e., the distribution of latent factors that we get from applying the encoder, $q(z)$, is not actually Gaussian as our generative model, $p(z)$, specified). For VAEs this is shown and discussed in the InfoVAE paper. They also show how to add a penalty so that $D_{MMD}(p(z), q(z))$ is minimized. Another approach that might be more powerful for getting true Gaussian latent factors is "normalizing flows". I'm not an expert on those, but one place to start is this paper.

  • $\begingroup$ VAE introduces additional distortion/blurring in q(z|x), what is unnecessary and doesn't seem to have any advantage (?) It is avoided in others from the list above - just enforcing Gaussian-like distribution in standard AE. Sure sampling is essential: that taking z ~ N(0,1) we get nearly uniform distribution in y ... but not having exactly Gaussian doesn't seem too dangerous - it seems we just need their somehow uniform covering (?) $\endgroup$
    – Jarek Duda
    Dec 10, 2018 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.