A prior for what?
Let's take a toy example of a Deep Neural Network with weights w being fit on a dataset D with an L2/Mean Square Error loss.
I looked up Maximum Likelihood Estimation (MLE) and Maximum a Posteriori Estimation (MAP) regarding the regularization and I think I have some vague intuition:
Take 1: The use of L2 loss in this setting induces the assumption that my likelihood P(D|w) is a Gaussian distribution and minimizing L2 then equates to maximizing the log of the likelihood. I think I got this right, but please correct me if I'm already wrong here. So L2 loss then somehow should in theory affect the distribution of the residuals and it also induces a normal distribution of the residuals (residuals being predicted output - ground truth label).
How do the distribution of the residuals relate to the likelihood? Are these two concepts even related? Distribution of residuals is a frequentist thing and likelihood is more like a bayesian thing.
Take 2: If I would like to find an MAP estimate, I have to choose a prior distribution for the weights, P(w). If I choose the normal distribution as the prior, the MAP estimate can be found by adding the L2-regularization of the weights to the loss function. Also if I understood correctly, if I choose the Uniform distribution as my prior, then I get back the MLE.
Is my intuition true? Also both MLE and MAP are point estimates. Is it somehow possible to sample (for instance with MC) from the full likelihood and full posterior probabilities?