8
$\begingroup$

Bayesian neural nets (BNN) are very popular topic. With development of variational approximation it became possible to train such models much faster then with Monte Carlo sampling. BNNs allow such interesting features as natural regularisation and even uncertainty estimation. So, the question is: why haven't we still completely migrated on BNNs?

I can assume that variational inference does not provide enough accuracy. Is it the only reason?

Improve this question
$\endgroup$
1
  • $\begingroup$ That's a good question, I believe most people are not aware of it, the math behind is harder, and the implementation examples scarce, if VI improves training I guess more people will be willing to give a try. $\endgroup$
    – Germán Alfaro
    Commented Apr 22, 2018 at 4:16

2 Answers 2

Reset to default
6
$\begingroup$

There are still many disadvantages of BNN compared with NN as listed below:

  1. The computational cost is heavier. In here I am not just referring to the cost of training, i.e. getting the posterior distribution of all parameters. This part is in fact OK if you use variational inference with a simple distribution family for BNN parameters. After your model is deployed and you want to make an inference, then you will need to sample N parameters from their posterior distribution in order to get the distribution of output, and this is N times more computational cost than just using NN.
  2. The tools of BNN is not popularized yet and is not so automatic as tools of NN.
  3. You need to make some assumptions about your prior, which is relatively difficult for most users.
  4. This is the most important reason that I think why BNN is not adopted universally instead of NN: the uncertainty we get is not as useful as we thought at first glance. Let's take an easy example: say you have two types of customers. Type A will have equal probability of giving you \$40 or \$60, and Type B will have equal probability of giving you \$30 or \$70. They have equal expectation, but larger uncertainty for the type A customer. Assume your BNN works perfectly well to tell one distribution from another. However, the uncertainty here does not matter if you have one million customers of each, because at that time what matters is not the uncertainty of individual customers, but the uncertainty of average, which goes towards zero when your number of customers goes larger according to law of large numbers. Therefore, you really do not need uncertainty in your model most of the time.
Improve this answer
$\endgroup$
2
  • $\begingroup$ In regression experiment with two-layer Bayesian net I found that it's not easy to get good uncertainty estimates. For example I had to choose specific model parametes to make the model uncertain far from training data, and if the model is too overparameterized there can be uncertainty spikes inside the space covered by training data. So this means that if you make a Bayesian net and optimize ELBO you still doesn't get desired uncertainty. Is it true? It also seems that uncertainty requires some uncertainty quality validation techniques. Are there some? $\endgroup$
    – Leonid
    Commented May 14, 2018 at 7:04
  • 1
    $\begingroup$ @Leonid Based on my very limited knowledge of Bayesian NN, I would say the uncertainty should only be taken qualitatively, because you might be using a normal distribution to approximate a very weird posterior distribution of parameters due to the very nonlinear nature of NN. As for the validation techniques, you can calculate the likelihood of your test data using the posterior distribution of parameters, and higher likelihood indicates a better model in terms of both the mean and variance of prediction. $\endgroup$
    – DiveIntoML
    Commented May 14, 2018 at 17:52
0
$\begingroup$

BNN is not capable to recognize posterior distribution. It returns expectation and variance (confidence interval in the assumption of normal distribution). It is overkill to build a stochastic model for modelling expectation and variance. You can find backup experiments on ezcodesample.com proving that it is completely fake.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.