I was reading Yarin gal's Phd thesis regarding the use of dropout as a technique to turn a neural network in a bayesian neural network. at page 32 of the pdf (18 for the thesis) he writes that the probability distribution for the output $y^* $ for a new input point $x^*$ can be obtained by integrating over the space of the parameters, i.e. the weights of the neural network:

$$ p(y^*|x^*,D)= \int_w p(y^*|x^*,w) p(w|D) \mathrm{d} w ;$$

and he calls this procedure simply inference. but what's the precise name of this procedure? some says bnns rely on model averaging, other sources says that they marginalize over the weight.

So what's the correct definition?

I think that BNNs approximate numerically the expected a posteriori estimator but i am not sure that's correct.

  • $\begingroup$ Bayesians call that distribution the posterior predictive distribution, so we might call the process of generating such a distribution "computing the posterior predictive [distribution]". $\endgroup$ May 24 at 15:31

1 Answer 1


I did not know how bayesian neural networks perform this operation prior to this answer, however I think the equation you mention is not specific to Bayesian neural networks but is indeed a generic Bayesian model averaging operation.

You can refer to Fragoso and Neto 2018 for an introduction and review of bayesian model averaging techniques.

The equation you state corresponds to equation (6) from this review (albeit with a, more generic, continuous formulation instead of a discrete one), and computes the marginal posterior distribution of a model output across all models. In your case the chosen output is the model's final prediction $y^*$, but you could perform this operation on different outputs (e.g intermediate embedding layers of a neural network)

It can indeed be termed inference (in the machine learning sense) since you are making a prediction on unseen data based on a set of previously trained models.

Regarding the Bayesian neural network, after a quick glance over the Jospin et al 2022 paper you mentioned I think the two following quotes answers your question:

A BNN is defined slightly differently across the literature, but a commonly agreed definition is that a BNN is a stochastic artificial neural network trained using Bayesian inference.

Stochastic neural networks are a type of ANN built by introducing stochastic components into the network. This is performed by giving the network either a stochastic activation (Figure 3b) or stochastic weights (Figure 3c) to simulate multiple possible models θ with their associated probability distribution p(θ). Thus, BNNs can be considered a special case of ensemble learning.

Thus the bayesian averaging performed for bayesian neural networks corresponds to averaging (e.g through MCMC) over the stochastic parameters of your network (the $w$ from your equation corresponds to activation functions or weights values). During training, instead of fixing precise values for your weights, you learn the parameters of the chosen distributions for the activation functions and weights (e.g expectation and variance), which results in an ensemble of possible neural networks. At inference time you sample this ensemble and perform bayesian averaging.

  • $\begingroup$ yes some authors says that's model averaging, but it's not the only way used to describe the process. other authors says that the strength of bnns lies in the marginalization of the weights. to me looks like they numerically compute the EAP estimator of the target variable y*... to an inexperienced user like me they all seems correct so I was wondering if they are equivalent ways to describe it or not. $\endgroup$
    – Alucard
    May 24 at 13:10
  • $\begingroup$ From what I understood both model averaging and marginalization over the weights stand for the same operation. If only the weights are stochastic then computing the marginal posterior distribution over models (model averaging) is exactly computing the marginal posterior distribution through marginalization over the weights. $\endgroup$ May 24 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.