Deterministic Decoder for Variational Bayes Autoencoder or RNN I was wondering if it is possible to use just deterministic function in the output layer of a variational Bayes autoencoder or RNN? Most of papers that I have read are using Gaussian Mixture Models (GMM) in the output layers but how about using just a MLP with TANH activation function in the output layer? I am talking about math not performance, is there any problem with that?  
 A: For the objective function of the VAE, the ELBO, you need the expected log likelihood terms: $\mathbb{E}\left [ \log p(x_i|z) \right]$ for data point $x_i$. If you want to use gradient-based optimisation, this part needs to be differentiable, which is the case for e.g. Gaussian or Bernoulli log-likelihoods.
If you don't want your decoder to be Gaussian or Bernoulli or Gamma or so, but a point mass instead, the density will be a dirac function. This function assigns an infinite likelihood to its location, and zero everywhere. And as such it is not differentiable which again rules out gradient-based techniques. These methods are called likelihood-free methods, and there exists a large body of work on them.
There are ways around it. One way is to do it exactly as GANs, through a ratio estimator parameterised as a neural network. A paper exploring such methods in the context of variational inference is:


*

*Tran, Dustin, Rajesh Ranganath, and David M. Blei. "Deep and hierarchical implicit models." CoRR, abs/1702.08896 (2017).

