Which likelihood and prior corresponds to which loss and regularization?

Question

In Bayesian linear regression having the objective $$\min_w \underbrace{\sum_{i=1}^N (w^Tx_i - y_i)^2}_{\text{log-likelihood}} + \underbrace{\lambda ~ w^Tw}_{\text{log prior distribution}}$$ can be seen this as the maximization of the log posterior distribution. The log posterior distribution then falls apart into two parts: A part corresponding to the log likelihood and a part corresponding to the log of the prior distribution.

In the above case we made the assumptions

(1) That the model is given by $y = w^Tx + \epsilon$ with $\epsilon$ being white noise, leading to the specific form of the log-likelihood (squared error).

(2) That the prior is given by a 0-centered Gaussian, leading to the specific form of the regularization term ($L_2$ regularization)

My Question is: Does anyone know anymore corresponding pairs like this (likelihood = loss function or prior = regularization)? Is there maybe some list of well known connections somewhere?

And more generally: Given an objective loss and regularization function, when can I find a fitting likelihood and prior distribution, s.t. the maximization of the posterior distribution corresponds to my objective.

Answer 1

1. Another such pair is Gaussian data generating process (which gives the same loss function as the one given in the question) along with Laplace Distribution as prior for the parameters. It gives the L1 norm regularizer.
2. As far as I know, there is no formal way to discover such links (quite like the conjugate prior for a distribution).

Answer 2

The prior appearing in the loss function is a general idea not limited to linear regression. It is simple to understand. Assume you do some Bayesian inference with a prior $p(\theta)$ and a likelihood $l(x,\theta)$. The the posterior is:

$$P(\theta,x)\propto p(\theta)l(x,\theta)$$

Assume you want to find the mode (maximum) of the posterior (or equivalently its $\log$) as an estimator for $\theta$. You can just write:

$$\log(P(\theta,x))= \log (p(\theta))+\log(l(x,\theta))+c$$

That's how you get a two part loss function. This works for any kind of model, not only linear regression. Another classical example is using a Laplace prior and this is known as $L^1$ regularization. Gaussian prior is $L^2$ regularization.

With a Gaussian prior and Gaussian model (linear regression), the posterior is Gaussian and thus the mode happens to be the same as the mean. That's why this "mode estimator" is especially natural in Bayesian linear regression.