I was looking through the literature on regularization, and often see paragraphs that links L2 regulatization with Gaussian prior, and L1 with Laplace centered on zero.

I know how these priors look like, but I don't understand, how it translates to, for example, weights in linear model. In L1, if I understand correctly, we expect sparse solutions, i.e. some weights will be pushed to exactly zero. And in L2 we get small weights but not zero weights.

But why does it happen?

Please comment if I need to provide more information or clarify my path of thinking.

  • $\begingroup$ Related: Why is Lasso penalty equivalent to the double exponential (Laplace) prior? $\endgroup$ – amoeba Nov 20 '15 at 17:48
  • 1
    $\begingroup$ A really simple intuitive explanation is that the penalty decreases when using an L2 norm but not when using an L1 norm. So if you can keep the model part of the loss function about equal and you can do so by decreasing one of two variables it's better to decrease the variable with a high absolute value in the L2 case but not in the L1 case. $\endgroup$ – testuser Aug 12 '16 at 0:05

The relation of Laplace distribution prior with median (or L1 norm) was found by Laplace himself, who found that using such prior you estimate median rather than mean as with Normal distribution (see Stingler, 1986 or Wikipedia). This means that regression with Laplace errors distribution estimates the median (like e.g. quantile regression), while Normal errors refer to OLS estimate.

The robust priors you asked about were described also by Tibshirani (1996) who noticed that robust Lasso regression in Bayesian setting is equivalent to using Laplace prior. Such prior for coefficients are centered around zero (with centered variables) and has wide tails - so most regression coefficients estimated using it end up being exactly zero. This is clear if you look closely at the picture below, Laplace distribution has a peak around zero (there is a greater distribution mass), while Normal distribution is more diffuse around zero, so non-zero values have greater probability mass. Other possibilities for robust priors are Cauchy or $t$- distributions.

Using such priors you are more prone to end up with many zero-valued coefficients, some moderate-sized and some large-sized (long tail), while with Normal prior you get more moderate-sized coefficients that are rather not exactly zero, but also not that far from zero.

enter image description here

(image source Tibshirani, 1996)

Stigler, S.M. (1986). The History of Statistics: The Measurement of Uncertainty Before 1900. Cambridge, MA: Belknap Press of Harvard University Press.

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), 267-288.

Gelman, A., Jakulin, A., Pittau, G.M., and Su, Y.-S. (2008). A weakly informative default prior distribution for logistic and other regression models. The Annals of Applied Statistics, 2(4), 1360-1383.

Norton, R.M. (1984). The Double Exponential Distribution: Using Calculus to Find a Maximum Likelihood Estimator. The American Statistician, 38(2): 135-136.

  • $\begingroup$ Wow, this is very good explanation, and also special thanks for linked question where the regularization norms are intuitively linked to mode, meadian and mean, this really clarifies a lot for me! $\endgroup$ – Dmitry Smirnov Oct 16 '15 at 11:09
  • 1
    $\begingroup$ @Tim, The Cauchy Distribution has Heavy Tail yet the probability for Zero is less than the Normal Distribution. So how come it induce sparse solution? $\endgroup$ – Royi Nov 9 '16 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.