Bayesian Linear Regression on the top of deterministic neural network

Question

I understand the concepts of Bayesian linear regression and regular neural networks separately, but I cannot wrap my head around how to combine both.

In a general setting, lets say I have a (deterministic) regular neural network, and I would like to build a Bayesian linear regression on the top of it and train them together. If it's not Bayesian linear regression, we can easily train them with backpropagation. But with two together, I do not understand how to train the neural network using gradient descent.

For my more specific problem setting, I have a set of vectors, and I want to first apply the same linear transformation to each of them, and then find the best linear combination of them to fit a target vector. The linear transformation is a matrix that maps the same dimension of the vectors to the same dimension. The linear combination part can be formulated as linear regression problem and the whole formula is as follows:

$\operatorname*{argmin}_{\theta, W} ||\theta^T(XW^T)-y||^2 $ where specifically, $X \in R^{m\times d}, W \in R^{d\times d}, \theta \in R^m, y \in R^d$ and $W$ is the transformation matrix and $\theta$ is the coefficient of the linear regression. I would like to pose prior on $\theta$ and thus make it Bayesian, but let $W$ remain deterministic.

Maybe it's because of my lack of understanding in Bayesian networks and other knowledge, I have searched up the internet but cannot find the right contents I want. I may be over-complicating things too. Please give me a direction or keywords that points to the solution of my problem. Thank you very much.

Answer 1

There's a simple option, if you are fine with maximum-a-posteriori estimation and prediction, but don't care so much about uncertainty. I guess you don't, because it's hard to see how you'd get that with the preceding "deterministic" neural network in the mix. In that case you can just penalize the parameters of your last linear layer as per the prior distributions you want (i.e. add that to the loss function).

If you want a fuller posterior (but are happy to ignore that the inputs are really also model estimates from the deterministic NN, the uncertainty about which does not end up being propagated properly), then perhaps look at O'Hagan 2010 or the like, where you can see how a Bayesian linear regression is just solving some linear algebra, too. That is, as long as you are fine to work with conjugate priors (or mixtures of them, which can approximate most things). I wonder whether you can convert this into the final layer of your neural network, but that's much more involved (no idea how complex this gets).

• O'Hagan, A., 2010. Kendall's Advanced Theory of Statistic 2B. John Wiley & Sons.