What does it mean in Deep Learning, that L2 loss or L2 regularization induce a gaussian prior?

Question

A prior for what?

Let's take a toy example of a Deep Neural Network with weights w being fit on a dataset D with an L2/Mean Square Error loss.

I looked up Maximum Likelihood Estimation (MLE) and Maximum a Posteriori Estimation (MAP) regarding the regularization and I think I have some vague intuition:

Take 1: The use of L2 loss in this setting induces the assumption that my likelihood P(D|w) is a Gaussian distribution and minimizing L2 then equates to maximizing the log of the likelihood. I think I got this right, but please correct me if I'm already wrong here. So L2 loss then somehow should in theory affect the distribution of the residuals and it also induces a normal distribution of the residuals (residuals being predicted output - ground truth label).

How do the distribution of the residuals relate to the likelihood? Are these two concepts even related? Distribution of residuals is a frequentist thing and likelihood is more like a bayesian thing.

Take 2: If I would like to find an MAP estimate, I have to choose a prior distribution for the weights, P(w). If I choose the normal distribution as the prior, the MAP estimate can be found by adding the L2-regularization of the weights to the loss function. Also if I understood correctly, if I choose the Uniform distribution as my prior, then I get back the MLE.

Is my intuition true? Also both MLE and MAP are point estimates. Is it somehow possible to sample (for instance with MC) from the full likelihood and full posterior probabilities?

Answer 1

I think you might be mixing up two ideas. First is that minimizing square loss is equivalent to maximum likelihood estimation of the network parameters (weights and biases) when the error term is assumed to be Gaussian. I think your reference is trying to convey this.

(Note that your residuals will be whatever they are and might turn out not to be even remotely normal. You don’t get to pick what the residual distribution will be.)

The second is that $L_2$ regularization on the network parameters is equivalent to putting a Gaussian prior on the parameters.

While both of these use Gaussian distributions for something, they are not the same.