Having looked through the internet and the paper, I find Bayes by Backprop very inaccessible for my intermediate understanding of variational inference.

Most online guides also lack some explaining like this: Weight Uncertainty in Neural Networks explains the entire ELBO idea, but then doesn't elaborate on why it is not just a variational autoencoder.

What is the difference between a variational autoencoder and Bayes By Backprop type neural net? Is it just effectively "encoder" of the VAE but regressed against y? If that is the case, how is that different from a Mixture Density type neural network?

I could look into the code, but my approach is to understand them separately so that I can stay critical about the implementations - and everyone can post any code so there is a reason to be critical.


1 Answer 1


A VAE is a latent variable model. The encoder estimates for each input, the corresponding posterior distribution $P(z|x)$ on the latent space $z$. The objective is typically to obtain a density model $P(x)$ of the data.

In Bayes by Backprop, the setting is that you want estimate the posterior distribution over the weights $P(\theta|D)$. This differs from a VAE because

  1. Every single data point $x$ corresponds to a different posterior $P(z|x)$ in a VAE -- and an encoder is used to estimate this posterior. On the other hand in BBB, there is only a single posterior distribution on the weights of the network, which isn't a direct function of any specific datapoint (of course it implicitly depends on the training data as a whole).

  2. The VAE in its most popular form opts for a gaussian prior and posterior, allowing easy computation of the KL term or "complexity cost". BBB opts to estimate this cost term by sampling, with the advantage of allowing more complex prior distributions (such as spike and slab).

A mixture density network isn't intrinsically bayesian in any way that I'm aware of.

  • $\begingroup$ It seems to me then that the main different between an MDN and a VAE is that there is no KL-penalty due to the absence of Bayesian interpretation, and also there is no decoder. VAE also uses neural networks to predict parameters of a Gaussian, but in the MDN case it is a likelihood while in the VAE it is a posterior. BBB on the other hand tries to then model some sort of "epismetic uncertainty" too, because it allows us to express our beliefs about the weights of the neural network itself? $\endgroup$
    – boomkin
    Jul 15, 2019 at 14:43
  • $\begingroup$ @boomkin a MDN defines some distribution $P(y|x)$, but it's a very general network because you can just insert an MDN output layer anywhere you want -- whether that's at the end of a VAE, or with a classification network you're training with BBB... etc. A VAE is a generative model. BBB is a model agnostic procedure for obtaining uncertainty on the weights of any neural network. You can apply BBB to any network -- even a VAE if you so desire. In fact you could train a VAE with an MDN output layer using BBB :) $\endgroup$
    – shimao
    Jul 16, 2019 at 1:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.