This post follows this one: Why does ridge estimate become better than OLS by adding a constant to the diagonal?

Here is my question:

As far as I know, ridge regularization uses a $\ell_2$-norm (euclidean distance). But why do we use the square of this norm ? (a direct application of $\ell_2$ would result with the square root of the sum of beta squared).

As a comparison, we don't do this for the LASSO, which uses a $\ell_1$-norm to regularize. But here it's the "real" $\ell_1$ norm (just sum of the square of the beta absolute values, and not square of this sum).

Can someone help me to clarify?

  • 2
    $\begingroup$ The penalty term in ridge regression is the squared L2 norm. See these slides written by Tibshirani as an example (slide 7) stat.cmu.edu/~ryantibs/datamining/lectures/16-modr1.pdf See also here en.wikipedia.org/wiki/Tikhonov_regularization $\endgroup$ – boscovich Oct 15 '14 at 13:57
  • $\begingroup$ Small point of clarification, these are slides from Ryan Tibshirani not Rob. $\endgroup$ – Ellis Valentiner Oct 15 '14 at 14:03
  • $\begingroup$ ok, thanks a lot for the clarification. But I don't understand why squared for L2 and not squared for L1. Don't we have a general formulae for any kind of regularization? $\endgroup$ – PLOTZ Oct 15 '14 at 14:05
  • $\begingroup$ @user12202013: thank you for pointing that out. I didn't notice that. $\endgroup$ – boscovich Oct 16 '14 at 11:16

Ridge and lasso are two ways of regularizing and a regression. Lasso regression imposes a constraint on the sum of absolute coefficients:

$ \sum_i \sqrt{\beta_i^2} = ||\beta||_1 $

Ridge regression imposes a constraint of the sum of squared differences:

$ \sum_i \beta_i^2 = \sqrt{\sum_i \beta_i^2}^2 = ||\beta_i||_2^2 $

You suggested to introduce even another norm, the euclidean length of the coefficients:

$ \sqrt{\sum_i \beta_i^2} = ||\beta_i||_2$

The difference between Ridge regression and the euclidean length is the squaring. This does change the interpretation of the regularization. While both ridge and the euclidean length regularize towards zero, ridge regression also differs the amount of regularization. Coefficients that are further from zero pull stronger towards zero. This makes it more stable around zero because the regularization changes gradually around zero. This is not the case for the euclidean length, or as a matter of fact, for lasso regression.

| cite | improve this answer | |

There are lots of penalized approaches that have all kinds of different penalty functions now (ridge, lasso, MCP, SCAD). The question of why is one of a particular form is basically "what advantages/disadvantages does such a penalty provide?".

Properties of interest might be:

1) nearly unbiased estimators (note all penalized estimators will be biased)

2) Sparsity (note ridge regression does not produce sparse results i.e. it does not shrink coefficients all the way to zero)

3) Continuity (to avoid instability in model prediction)

These are just a few properties one might be interested in a penalty function.

It is a lot easier to work with a sum in derivations and theoretical work: e.g. $||\beta||_2^2=\sum |\beta_i|^2$ and $||\beta||_1 = \sum |\beta_i|$. Imagine if we had $\sqrt{\left(\sum |\beta_i|^2\right)}$ or $\left( \sum |\beta_i|\right)^2$. Taking derivatives (which is necessary to show theoretical results like consistency, asymptotic normality etc) would be a pain with penalties like that.

| cite | improve this answer | |
  • $\begingroup$ ok, thanks. But why squared for L2 and not squared for L1? Don't we have a general formulae for any kind of regularization? This is puzzling me... $\endgroup$ – PLOTZ Oct 15 '14 at 14:34
  • $\begingroup$ @PLOTZ I added a bit to my answer. $\endgroup$ – bdeonovic Oct 15 '14 at 17:19
  • $\begingroup$ Thanks a lot Benjamin! For sure it's clearer now! I didn't get this theoretical purpose before your answer. Many thanks for your answer. $\endgroup$ – PLOTZ Oct 16 '14 at 5:46
  • $\begingroup$ @Benjamin: in point #1 did you actually mean "(not all penalized estimators will be unbiased)"? Ridge regression –just to name one– is biased. $\endgroup$ – boscovich Oct 16 '14 at 7:40
  • $\begingroup$ whoops yes thanks for catching that! I think in fact all penalized estimators will be biased. $\endgroup$ – bdeonovic Oct 16 '14 at 11:37

Actually both the square of the $\ell_2$-norm and the $\ell_1$-norm come from a same class of regularization: $\|\boldsymbol{\beta}\|_p^p$ when $p > 0$.

The Ridge regression is then using $p=2$, and the Lasso $p=1$ but one can use other values of $p$.

For example you have sparse solution for all values of $p \leq 1$, and the smaller the value of $p$ the sparser the solution.

For values of $p \leq 1$ your objective is no more smooth so the optimization become harder; for $p<1$ the objective is non-convex and so the optimization even harder...

| cite | improve this answer | |

I believe there is an even simpler answer here, although "why" questions are always hard to answer when a technique is developed. The squared $l_2$-norm is used so that the regularization term is easily differentiable. Ridge regression minimizes:

$$\|\mathbf{y - X\beta}\|^2_2+\lambda\|\beta\|_2^2$$

Which can also be written: $$\|\mathbf{y - X\beta}\|^2_2+\lambda\beta^T\beta$$

This can now be easily differentiated wrt $\beta$ to get the closed-form solution:

$$\hat\beta^{\text{ridge}} = (\mathbf{X}^T\mathbf{X} + \lambda I)^{-1}\mathbf{X}^T\mathbf{y}$$

from which all kinda of inference can be derived.

| cite | improve this answer | |

Consider one other important difference between using the square of the $\ell_2$ norm (i.e. ridge regression) and the unmodified $\ell_2$ norm: the derivative of the $\ell_2$ norm of $x$, $||x||_2$, at $x$ is given by $\frac{x}{ ||x||_2}$ and therefore not differentiable at the zero vector. That is, although the $\ell_2$ norm does not do individual variable selection like the lasso, it could theoretically yield $\beta=0$ as the solution to the maximum penalized likelihood. By squaring the $\ell_2$ norm in the penalty, the ridge-type penalty is differentiable everywhere and can never yield such a solution.

This behavior is exactly (by my understanding) why the group lasso (Yuan and Lin) and the sparse group lasso (Simon, et al.), etc, use the $\ell_2$ norm (on prespecified subsets of the coefficients) instead of the square of the $\ell_2$ norm.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.