What is the idea behind Bayes By Backprop? Having looked through the internet and the paper, I find Bayes by Backprop very inaccessible for my intermediate understanding of variational inference.
Most online guides also lack some explaining like this: Weight Uncertainty in Neural Networks explains the entire ELBO idea, but then doesn't elaborate on why it is not just a variational autoencoder.
What is the difference between a variational autoencoder and Bayes By Backprop type neural net? Is it just effectively "encoder" of the VAE but regressed against y? If that is the case, how is that different from a Mixture Density type neural network?
I could look into the code, but my approach is to understand them separately so that I can stay critical about the implementations - and everyone can post any code so there is a reason to be critical.
 A: A VAE is a latent variable model. The encoder estimates for each input, the corresponding posterior distribution $P(z|x)$ on the latent space $z$. The objective is typically to obtain a density model $P(x)$ of the data.
In Bayes by Backprop, the setting is that you want estimate the posterior distribution over the weights $P(\theta|D)$. This differs from a VAE because


*

*Every single data point $x$ corresponds to a different posterior $P(z|x)$ in a VAE -- and an encoder is used to estimate this posterior. On the other hand in BBB, there is only a single posterior distribution on the weights of the network, which isn't a direct function of any specific datapoint (of course it implicitly depends on the training data as a whole). 

*The VAE in its most popular form opts for a gaussian prior and posterior, allowing easy computation of the KL term or "complexity cost". BBB opts to estimate this cost term by sampling, with the advantage of allowing more complex prior distributions (such as spike and slab).
A mixture density network isn't intrinsically bayesian in any way that I'm aware of.
