# Bayes by Backprop applied to Regression

I have been reading the Bayes by Backprop paper "Weight Uncertainty in Neural Networks" from 2015 (https://arxiv.org/pdf/1505.05424.pdf). I think I have a decent understanding of the content of the paper up until the part where it is applied to a regression problem. My issue is that to do the backpropagation we have to do computations involving the likelihood term $$P(\mathcal{D}|w)$$ where $$\mathcal{D}$$ is the data. This is based on the model specified by our network so in the classification case I believe this expression for any of our data points is just the probability predicted by the forward pass through the network.

After working on MNIST as a classification example in the paper (which I believe I understand assuming the above is correct) the authors move on to a regression example where they just generate some data from a simple function and attempt to fit it with their method and an ordinary neural network. They mention that the two networks are fit by minimising a conditional Gaussian loss and I'm not sure what is meant by this. The only thing that seems reasonable to me is that in the case of both networks the underlying model is that $$Y|_X$$ is Gaussian. I believe this would mean in the ordinary network we are treating the network output as an estimate for the mean of $$Y|_{X_i}$$ and therefore we use a squared error loss for training. In the Bayes by Backprop case then we would want $$P(\mathcal{D}|w)$$ to correspond to the likelihood for the target value so this would be the likelihood for a Gaussian with mean $$\hat{Y}_i$$ evaluated at the data $$(X_i, Y_i)$$, where $$\hat{Y}_i$$ is the value predicted by the network.

My first two questions are as follows:

• Is my understanding set out above correct?

• For the Bayes by Backprop regression above, we have no variance specified but we need to calculate the likelihood. How is this dealt with?

The results for this part are displayed by plotting the fits with confidence bands. I'm unsure on exactly how the confidence bands are being obtained. Firstly for the Bayesian approach, the result of fitting the network is a posterior probability distribution for each weight so the distribution for $$\hat{Y}|_X$$ is determined by these, however I'm not sure how practical it would be to use this to compute confidence bands due to fact that this distribution will generally have a complicated dependence on the weight distributions. Since prediction is achieved by sampling weights and averaging multiple predictions $$\hat{Y}$$ maybe it would make more sense to use these predictions to compute a confidence band too, but then we would need to make some sort of assumption on the posterior for $$\hat{Y}|_X$$ or justify using the central limit theorem. Secondly for the ordinary neural network there are also confidence regions on the plot. I'm not sure where these come from as my motivation for why this Bayesian approach was important is because on its own an ordinary neural network doesn't give us error estimates. Based on these points then:

• What exactly are the displayed confidence bands for the Bayes by Backprop network fit? Is it one of the two approaches I mentioned?
• How are confidence regions being obtained for the ordinary neural network? I was wondering if this is linked to the second of my first two questions since that involved specifying a variance for $$Y|_X$$ but that would be constant and the bands in the plot are not of constant width.
• This website works best if you ask one question at a time. Feel free to post each of these as separate questions. Mar 31, 2021 at 0:24
• They're all related to one subsection of the same paper, do they still need to be separated? They would probably each have to almost the full length to explain the context. Mar 31, 2021 at 0:43
• It’s helpful for getting faster answers. Several people may have the time or expertise to answer one part, but not the complete question. A partial answer to the uber-question doesn’t properly qualify as an answer here. Mar 31, 2021 at 0:49

I agree the authors didn't do a great job of clarifying this. My best guess is that the regression model looks like: $$y_i \sim \mathcal{N}(\mu=f(x_i), \sigma^2 = g(x_i))$$ where $$f$$ and $$g$$ are neural networks (or more precisely, probably two branches of the same neural network).
1. sample $$\theta$$ from the variational posterior,
2. given $$x$$, sample $$y$$ according to the above model,
4. compute the 25 and 75% percentile of the sampled $$y$$.
• I have seen other approaches where something like the network producing estimates for both $\mu$ and $\sigma^2$ is done, so I guess that seems likely. Apr 2, 2021 at 14:28
• My comment on the confidence bands was based on the fact that the distribution of $y$ for a given $x$ is also determined by this method (as well as the weights distributions). As a result I'm not sure why we would use sampling of $y$ to come up with a confidence region when we actually know the form of the distribution for $y$ so we don't need to do this empirically. Apr 2, 2021 at 14:34
• you know the conditional $p(y|x,\theta)$ but not the marginal $p(y|x)$. so either you integrate (intractable), or you just sample, which is by far easier Apr 2, 2021 at 16:37
• @EGN123 could you elaborate on that? where have you seen this? i'm looking for a way to apply bayes by backprop to regression and having estimates for both $\mu$ and $\sigma^2$ is exactly the outcome i want. Jul 1, 2021 at 17:10