I am currently reading about variational autoencoders.
Some of the papers I've read are:
- Tutorial on Variational Autoencoders by Doersch: https://arxiv.org/abs/1606.05908
- Auto-Encoding Variational Bayes by Kingma & Welling: https://arxiv.org/abs/1312.6114
- Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications - Haowen Xu et.al.: https://arxiv.org/abs/1802.03903
The papers differ in one fundamental issue, Doersch only has one layer which produces the standard deviation and mean of a normal distribution, which is located in the encoder, whereas the other have two such layers, one in exactly the same position in the encoder as Doersch and the other one in the last layer, before the reconstructed value.
I have seen the variant with only one such layer in some machine learning books as well.
So which one of the two methods is used more often? Is one superior to the other? What is the philosophy in using one vs using two such layers? (A good scheme can be found in the last paper on page 3)
This graphic is from the last paper I linked. It shows the typical latent variable layer, produced by sampling from a normal distribution. The $\mu_z$ and $\sigma_z$ are generated by the neural network. The first paper I mentioned only uses the latent layer in the encoding part, meaning the right part of the graphic, the decoder, does not have a sampling and does not produce $\mu_x$ and $\sigma_x$, but instead just recreates the input just by a normal feedforward network.
So to repeat my question: What are advantages/disadvantages of the the two approaches.