Approximating the posterior and learning the distribution over the weights after training

Question

I am familiar with the methods in variational inference in which after training we have access to the distribution over the network's weights. This is necessary for estimating epistemic uncertainty. Other than variational methods, we can benefit from sampling methods (MCMC) and ensembling as well. MC-dropout is another successful method that has been introduced in recent years. These methods make it possible to predict both epistemic and aleatoric uncertainties. What I would like to know is that do these algorithms (methods other than variational inference) also learn a distribution over the weights after the training phase is over? I am new to the field and so far have been able to read on variational inference papers but the above question lingers on the back of my mind so I thought maybe I can ask about it here. Any explanation or references to interesting sources will be immensely appreciated.

Answer 1

If you train Bayesian neural networks using back propagation(Bayes by backprop) we cannot backpropagate through a stochastic node, then we use the reparameterization trick.

For the MC dropout, any variable follow a this distribution(the 1 and 0 mask):

Here are some useful materials:

1. Hands-on Bayesian Neural Networks - a Tutorial for Deep Learning Users.
2. Probabilistic Deep Learning: with Python, Keras and Tensorflow Probability