# What is the idea behind Bayes By Backprop?

Having looked through the internet and the paper, I find Bayes by Backprop very inaccessible for my intermediate understanding of variational inference.

Most online guides also lack some explaining like this: Weight Uncertainty in Neural Networks explains the entire ELBO idea, but then doesn't elaborate on why it is not just a variational autoencoder.

What is the difference between a variational autoencoder and Bayes By Backprop type neural net? Is it just effectively "encoder" of the VAE but regressed against y? If that is the case, how is that different from a Mixture Density type neural network?

I could look into the code, but my approach is to understand them separately so that I can stay critical about the implementations - and everyone can post any code so there is a reason to be critical.

A VAE is a latent variable model. The encoder estimates for each input, the corresponding posterior distribution $$P(z|x)$$ on the latent space $$z$$. The objective is typically to obtain a density model $$P(x)$$ of the data.
In Bayes by Backprop, the setting is that you want estimate the posterior distribution over the weights $$P(\theta|D)$$. This differs from a VAE because
1. Every single data point $$x$$ corresponds to a different posterior $$P(z|x)$$ in a VAE -- and an encoder is used to estimate this posterior. On the other hand in BBB, there is only a single posterior distribution on the weights of the network, which isn't a direct function of any specific datapoint (of course it implicitly depends on the training data as a whole).
• @boomkin a MDN defines some distribution $P(y|x)$, but it's a very general network because you can just insert an MDN output layer anywhere you want -- whether that's at the end of a VAE, or with a classification network you're training with BBB... etc. A VAE is a generative model. BBB is a model agnostic procedure for obtaining uncertainty on the weights of any neural network. You can apply BBB to any network -- even a VAE if you so desire. In fact you could train a VAE with an MDN output layer using BBB :) Jul 16, 2019 at 1:34