In Bayesian probability, we think of having a prior distribution over a set of possible models, and updating this distribution every time we find new information.
But with a neural network with gradient descent, we instead have a single model at any point in time, and "infinitessimally" change this model with each new piece of information.
How can we interpret this approach from a Bayesian perspective? i.e. can we see a neural network with gradient descent as an approximation to Bayesian updating? Is there a good intuitive article written on this?
EDIT: my question is not "what is the difference between NN with SGD and bayesian methods?" My question is: "how can we interpret NN with SGD from a bayesian perspective (e.g. as an approximation)?