1
$\begingroup$

I wish to use a Variational autoencoder (VAE) as a generator for a multivariate distribution which originates from a graphical model - e.g. samples from a Bayesian Network (I have my reasons...).

I wanted to start with some toy problems, but got stuck on the very first one - getting a VAE to regenerate a mixture of Gaussians distribution. I generated a synthetic dataset of 16k samples from a mixture of 5, mostly non-overlapping, 2D gaussians, and trained a VAE to recreate this distribution (Tried many different architectures, all modeled after this official Pytoch VAE example with FC layers. Losses were MSE between the input and reconstructed input plus +$D_{KL}$ between the encoder's output and the prior).

The train loss plateaus pretty fast, no matter what the learning rate is, and later at test time when feeding standard normal noise to the decoder, the output looks nothing like the original mixture.

Question: Before I start a more serious debugging and/or architecture optimization effort than I already have, I'd like to know if my task is somehow ill defined or if this is expected to work?

Training set - 16k 2D samples drawn from a mixture of 5 equiprobable uncorrelated (diagonal covariance matrices) Gaussians enter image description here

Test set - 64 samples from a standard normal distribution passed through the decoder enter image description here

$\endgroup$
4
  • $\begingroup$ image of output? $\endgroup$
    – shimao
    Jan 16, 2020 at 4:12
  • $\begingroup$ @shimao I don't understand the question $\endgroup$ Jan 16, 2020 at 7:42
  • $\begingroup$ you say that the output looks nothing like the original mixture. can you attach an image showing this? $\endgroup$
    – shimao
    Jan 16, 2020 at 7:47
  • $\begingroup$ @shimao - Added. $\endgroup$ Jan 16, 2020 at 11:19

1 Answer 1

1
$\begingroup$

That looks like a bug. I tweaked the same repo, and used the standard toy "8-gaussians" dataset, and trained a VAE.

enter image description here enter image description here

Red points: samples from the 8-gaussians dataset

Blue points: sampled from the trained model

It's nowhere near perfect (and 8-gaussians is actually a non-trivial task for many other generative models), but the mass is concentrated in the right places, roughly speaking.

$\endgroup$
3
  • $\begingroup$ Much obliged. Do you mind sharing your code? $\endgroup$ Jan 19, 2020 at 8:36
  • $\begingroup$ Also, "8-gaussians is actually a non-trivial task for many other generative models". Can you provide a source or briefly explain why this is so? I wouldn't have expected this to be difficult for a deep generative model $\endgroup$ Jan 19, 2020 at 8:39
  • $\begingroup$ here: gist.github.com/ricsonc/34189cded625f59614fbafb4080a69f4 my intuition for why this is so difficult is that deep generative models are designed to learn some smooth and connected manifold in very high dimensional space, which is substantially different to the task of learning some "disconnected" distribution in low-dimensional space $\endgroup$
    – shimao
    Jan 19, 2020 at 23:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.