Comparison of Bayesian Neural Network with Multilayer Perceptron

Question

I have a machine learning project with not so much data, so I have the following reasons to use Bayesian neural network (not Bayesian network / directed graphical models) for my work:

1. There are indeed some nonlinear interaction between features in my model, so I do not want to use linear regression. And as I tried, xgboost does not work well for this dataset.
2. I heard Bayesian neural network is more tolerance of small dataset than multilayer perceptron.
3. I can get a free estimate of the uncertainty of my prediction.

However, I have the following question about Bayesian neural network as I have no experience with it:

1. Is it true that Bayesian neural network work better than multilayer perceptron? I synthesized some data using a function I chose, then gradually reduce the number of data gradually and measure the mean squared error from both model. Up until the point when the number of data is too small for both models, multilayer perceptron always perform better than Bayesian neural network.
2. How should I compare the performance of these two models? For my regression task, I used mean squared error for multilayer perceptron. But how about Bayesian neural network? Shall I use the MAP value of the parameters and report the corresponding mean squared error? Or shall I sample parameters multiple times and report the average mean squared error?

Answer 1

1. Well, in Bayesian NN you need to set a prior over your NN weights. Setting that properly is needed for it to deliver good performance. Also, if this is a full Bayesian setting where prediction is done under the the distribution over the parameters, corresponding to, $$P(y|x) = \int P(y|x,\theta)P(\theta|D)d\theta$$ then it also depends how you approximate that integral.

2. No, if it is full Bayesian and not just MAP, you need to report an expectation over your prediction. Which is to draw samples from the posterior distribution parameter and report the expectation (i.e., average) over the prediction.