Autoencoders want to minimize distortion of encoding-decoding process, preferably alongside evaluation by discriminant.
Generative autoencoders additionally would like latent variable from a chosen probability distribution, usually (multivariate) Gaussian.
While optimization to get sample looking like from Gaussian might sound simple, it isn't - it would be great to understand what do we really want here?
Many approaches were recently proposed, for example:
- The original Variational AutoEncoder (VAE) directly tested only KL for separate points – not ensuring their uniform coverage.
- A year ago there was proposed using Wasserstein metric (WAE, SWAE) to optimize distance between the entire sample and multivariate Gaussian. However, due to difficulty of calculation, it represents the distribution with a random sample, then still requires approximation.
- Half a year ago there was finally proposed a non-random analytic formula: by calculating L2 distance between 1D projections of Gaussian-smoothened sample, and averaging over all projection directions (CWAE). But still this is only guessing that we will get the desired distribution – tested usually only by verifying two moments (Marida test), still leaving huge freedom for continuous distributions.
- We can also directly optimize empirical distribution function to agree with CDF of Gaussian distribution, especially for radii $\|x_i\|$ and distances $\|x_i -x_j\|$.
As there are many ways to approach this problem, it would be great to specify it - understand what do we want to achieve here?
What do we really mean by "latent variables from Gaussian distribution"? How important it is?