Learning normal distribution with VAE

I am trying to use a Variational Autoencoder to learn a multivariate normal distribution. I know that from practical point of view this is pointless, as we can sample from a normal distribution itself, however I wanted to try this before going to more interesting applications.

I am using the algorithm presented here. Instead of training in the MNIST data, I sample from numpy's normal multivariate and map the result to $[0,1]$ using a sigmoid. I use this data to train the VAE.

After training I sample from a $\mathcal{N}(0,I)$ in the latent space and use the decoder to generate data. I expected that if I apply an inverse sigmoid to the generated data, I should get normally distributed data with the same mean and covariance as the dataset I used for training. I compare the training and the generated data using a scatter plot and I cannot get them to match.

My main question is whether this scheme can work (which means that I am just doing something wrong in the program) or if it is not correct to use VAEs this way at all (I just learnt about VAEs yesterday, so I am not that sure!).

Moreover, in the link I gave a Bernoulli distribution is used to calculate reconstruction loss. I tried both with this but also with a Gaussian:

reconstr_loss = 0.5 * tf.reduce_sum((self.x - self.x_reconstr_mean)*(self.x - self.x_reconstr_mean) + np.log(2*np.pi*self.sigma_hyper * self.sigma_hyper), 1)

where sigma_hyper is just a hyperparameter (I set it 1). Both ways didn't work.

Thanks a lot!

Edit: Some of the scatter plots. Left is a 2D multivariate normal (used for training - axes correspond to the two random variables) and right the data generated from the decoder. Both mean and covariance is apparently wrong in the generated data. • There are numerous reasons that lead to this kind of behaviour: i) optimisation did not work, ii) statistical modelling is wrong, ii) architecture of the network is wrong, iii) data is weird. I suggest you try if it can work on an even simper problem, like learning a 2D Gaussian with non-diagonal covariance and a linear decoder, which should suffice. Also, use a Gaussian log-likelihood for continuous data. – bayerj Oct 7 '17 at 19:08
• I tried 2D Gaussian with covariance [[1,-1],[-1,1]], one latent variable and linear decoder and it seems to work. I haven't calculated anything quantitative for the generated data (like mean or covariance) but from the scatter plots they seem ok. It also seems to work even if the encoder is linear as well. The same architecture with one latent variable obviously doesn't work for diagonal covariance. However if I use two latent variables (that is same as the input) then it works for diagonal covariance as well. – stavros11 Oct 9 '17 at 21:41
• The problem starts when I add hidden layers to encoder and decoder, as in the blog I referenced in my first post, where we have two hidden layers to each network. When I do this, it cannot even learn the Gaussian with non-diagonal covariance. I can check that the training dataset has zero mean, while the generated data have non-zero. Therefore, I guess that the problem is in the architecture. According to the blog, the same program works for the MNIST dataset, though. – stavros11 Oct 9 '17 at 22:24