Diversity of generated samples in VAE In a variational autoencoder (VAE), it is possible to generate new samples (i.e. images) based on the latent space. After having read quite a few papers about VAE, I still wonder what drives the diversity of generated (hallucinated) samples? Put differently, what can be done in a VAE to generate samples that are more diverse besides having more data to train on? 
 A: 
I still wonder what drives the diversity of generated (hallucinated) samples?

In the "ideal" case, every sample from the data set can be 
(a) encoded into the latent space such that it can be well-reconstructed and
(b) is "sufficiently close" to the origin that (depending on the empirical likelihood of that data sample) we are likely to "get" that point in latent space by sampling $z\sim\mathcal{N}(0,I)$. Note that (a) and (b) are somewhat in conflict (see the $\beta$-VAE), since keeping latent points far apart makes (a) easier but harms (b).
So I guess you could say that the diversity of the generated outputs is controlled by (1) the diversity of the data itself and (2) how well the marginal distribution $\int q(z|x) q(x) dx$ of the inference model matches the generative prior $p(z)$. (Alternatively put, (2) means that if you encode some input $x$ to some crazy value $z$ that is far from the origin, then you will never draw that point (or nearby it) as a sample, thus decreasing "diversity").

Put differently, what can be done in a VAE to generate samples that are more diverse besides having more data to train on?

Well, besides more data or more powerful networks, one way that comes to mind is to make the prior $p(z)$ more expressive and/or make the variational inference distribution more powerful. This is because it softens the conflict between (a) and (b) above, so it is easier for the model to make all the samples it has seen be "sample-able".
E.g., check out this work.
Perhaps in some cases it can also help to use adversarial losses in conjunction with the VAE. This is because classical likelihood losses care only about reconstruction in the data space (e.g., pixel-wise losses). 
This can be stifling in that the stochastic generator of a VAE cannot really deviate from exactly what it is given or it will be penalized.
Adversarial losses only care that the input could have been reasonably sampled from the empirical distribution, which lets the generator "experiment" as long as the output "looks reasonable". 
Here are some references on combining the two: 
[1],
[2],
[3],
[4],
[5],
[6].
However, training VAE-GANs is difficult: e.g., this work or this one.
