12
$\begingroup$

Recently I read some papers about the Bayesian neural network (BNN) [Neal, 1992], [Neal, 2012], which gives a probability relation between the input and output in a neural network. Training such a neural network is through MCMC which is different from the traditional back-propagation algorithm.

My question are: What is the advantage of using such a neural network? More specifically, could you provide some examples which better fit BNN rather than NN?

$\endgroup$
8
$\begingroup$

Bayesian neural nets are useful for solving problems in domains where data is scarce, as a way to prevent overfitting. They often beat all other methods in such situations. Example applications are molecular biology (for example this paper) and medical diagnosis (areas where data often come from costly and difficult expiremental work). Actually, Bayesian nets are univerally useful and can obtain better results for a vast number of tasks but they are extremely difficult to scale to large problems.

$\endgroup$
  • 2
    $\begingroup$ Can you expand on why Bayesian nets are difficult to scale? $\endgroup$ – Ellis Valentiner Mar 17 '15 at 15:38
6
$\begingroup$

One advantage of the BNN over the NN is that you can automatically calculate an error associated with your predictions when dealing with data of unknown targets. With a BNN, we are now doing Bayesian inference. Let's define our BNN prediction as $\bar{f}(x′|x,t)=∫f(x′,ω)p(ω|x,t)dω$, where $f$ is the NN function, $x'$ are your inputs, $ω$ are the NN parameters, and x,t are the training inputs and targets. This should be compatible with the syntax used by Neal in the links provided by @forecaster. Then we can calculate a standard deviation of the posterior predictive distribution, which I would naively use as an accuracy on the prediction : $\sigma(x′)=\sqrt{∫[f(x′,ω)−\bar{f}(x′|x,t)]^2p(ω|x,t)dω}$

$\endgroup$
  • 1
    $\begingroup$ This is an interesting addition to the conversation, but it is a bit short by our standards. Could you elaborate a bit and perhaps include a reference? $\endgroup$ – Sycorax May 5 '16 at 21:00
  • $\begingroup$ Sure. With a BNN, we are now doing Bayesian inference. Let's define our BNN prediction as $\bar{f}(x'|x,t) = \int{f(x',\omega)p(\omega|x,t) d\omega}$, where f is the NN function, x' are your inputs, $\omega$ are the NN parameters, and $x,t$ are the training inputs and targets. This should be compatible with the syntax used by Neal in the links provided by @forecaster. Then we can calculate a standard deviation of the posterior predictive distribution, which I would naively use as an accuracy on the prediction : $\sigma(x') = \sqrt(\int{[f(x',\omega)-\bar{f}(x'|x,t)]^2p(\omega|x,t) d\omega})$ $\endgroup$ – Michelle K May 6 '16 at 23:24
  • $\begingroup$ Please edit this into your answer. $\endgroup$ – Sycorax May 7 '16 at 0:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.