How can we interpret a neural network with sgd from a Bayesian perspective?

Question

In Bayesian probability, we think of having a prior distribution over a set of possible models, and updating this distribution every time we find new information.

But with a neural network with gradient descent, we instead have a single model at any point in time, and "infinitessimally" change this model with each new piece of information.

How can we interpret this approach from a Bayesian perspective? i.e. can we see a neural network with gradient descent as an approximation to Bayesian updating? Is there a good intuitive article written on this?

EDIT: my question is not "what is the difference between NN with SGD and bayesian methods?" My question is: "how can we interpret NN with SGD from a bayesian perspective (e.g. as an approximation)?

Answer 1

I think you would be interested in "Stochastic Gradient Descent as Approximate Bayesian Inference" by Stephan Mandt, Matthew D. Hoffman, David M. Blei.

Stochastic Gradient Descent with a constant learning rate (constant SGD) simulates a Markov chain with a stationary distribution. With this perspective, we derive several new results. (1) We show that constant SGD can be used as an approximate Bayesian posterior inference algorithm. Specifically, we show how to adjust the tuning parameters of constant SGD to best match the stationary distribution to a posterior, minimizing the Kullback-Leibler divergence between these two distributions. (2) We demonstrate that constant SGD gives rise to a new variational EM algorithm that optimizes hyperparameters in complex probabilistic models. (3) We also propose SGD with momentum for sampling and show how to adjust the damping coefficient accordingly. (4) We analyze MCMC algorithms. For Langevin Dynamics and Stochastic Gradient Fisher Scoring, we quantify the approximation errors due to finite learning rates. Finally (5), we use the stochastic process perspective to give a short proof of why Polyak averaging is optimal. Based on this idea, we propose a scalable approximate MCMC algorithm, the Averaged Stochastic Gradient Sampler.

Some additional relevant literature:

• Radford M. Neal was active in Bayesian inference and neural networks in the "first wave" of NN research in the 1990s. He published several articles and the book Bayesian Learning for Neural Networks.

• Sergios Theodoridis' Machine Learning: A Bayesian and Optimization Perspective is a truly massive tome (about 1000 pages) and includes a chapter on neural networks and deep learning.

• This paper crossed my desk recently: "Bayesian Neural Networks" by Vikram Mullachery, Aniruddh Khera, Amir Husain

  This paper describes and discusses Bayesian Neural Network (BNN). The paper showcases a few different applications of them for classification and regression problems. BNNs are comprised of a Probabilistic Model and a Neural Network. The intent of such a design is to combine the strengths of Neural Networks and Stochastic modeling. Neural Networks exhibit continuous function approximator capabilities. Stochastic models allow direct specification of a model with known interaction between parameters to generate data. During the prediction phase, stochastic models generate a complete posterior distribution and produce probabilistic guarantees on the predictions. Thus BNNs are a unique combination of neural network and stochastic models with the stochastic model forming the core of this integration. BNNs can then produce probabilistic guarantees on it's predictions and also generate the distribution of parameters that it has learnt from the observations. That means, in the parameter space, one can deduce the nature and shape of the neural network's learnt parameters. These two characteristics makes them highly attractive to theoreticians as well as practitioners. Recently there has been a lot of activity in this area, with the advent of numerous probabilistic programming libraries such as: PyMC3, Edward, Stan etc. Further this area is rapidly gaining ground as a standard machine learning approach for numerous problems

Answer 2

The Bayesian approach to probabilistic problems often uses graphical models like:

$X \sim F(Y)$

$Y \sim G(Z)$

$Z \sim H$

Where one set of observations has a distribution conditional on another.

This is most$\dagger$ of what you need for neural networks (survey). For example the model above could describe a single layer network where $Y$ was the activation in the hidden layer, $X$ the observations, and $Z$ a latent class. We'd just be 'hiding' some non-linear transformations in $F$, $G$ etc.

It's worth bearing in mind that neural networks $\neq$ gradient descent. Gradient descent is just a particularly effective way of fitting the models. A Bayesian inference technique which leverages the same tools is Variational Bayes. Although there are sampling approaches (HMC, SGLD etc etc.) that can also make use of gradient information.

So the difference is that in a Bayesian approach you also get some information about the distribution of your model parameters in addition to a point estimate of them. This could help, e.g. in problems where a peturbation to the parameters would cause a reclassification of an observation which would affect your decision making.

$\dagger$ the question arises of how to do things like max pooling, drop out and so forth. Many of these operations have Bayesian analogues in the form of particular distributions or transformations.

Answer 3

The simple, almost generic, answer to such question is that the difference is that Bayesians define models in probabilistic terms, so the parameters are considered as random variables and they assume priors for such parameters.

In classical setting, your model has some parameters and you are using some kind of optimization to find such parameters that best fit your data. In neural networks this is done almost exclusively by starting with some randomly initialized parameters and then using some variant of gradient descent to update the parameters, so that the loss is minimized.

In Bayesian setting, instead of point estimates of the parameters, we want to estimate the distributions of the parameters, because we consider them to be random variables. This is done by starting with some a priori distributions assumed for the parameters, that are updated using Bayes theorem when confronted with data. This is usually done either with some kind of optimization, or by Markov Chain Monte Carlo sampling (simulating draws from the posterior distribution).

So gradient descent is an optimization algorithm, while Bayesian approach is about defining the models differently (see also this comparison of maximum likelihood and gradient descent). Using gradient descent to minimize some kind of loss function does not have much to do with Bayesian approach because it does not consider any priors.