13
$\begingroup$

This post follows this one: Why does ridge estimate become better than OLS by adding a constant to the diagonal?

Here is my question:

As far as I know, ridge regularization uses a $\ell_2$-norm (euclidean distance). But why do we use the square of this norm ? (a direct application of $\ell_2$ would result with the square root of the sum of beta squared).

As a comparison, we don't do this for the LASSO, which uses a $\ell_1$-norm to regularize. But here it's the "real" $\ell_1$ norm (just sum of the square of the beta absolute values, and not square of this sum).

Can someone help me to clarify?

$\endgroup$
  • 2
    $\begingroup$ The penalty term in ridge regression is the squared L2 norm. See these slides written by Tibshirani as an example (slide 7) stat.cmu.edu/~ryantibs/datamining/lectures/16-modr1.pdf See also here en.wikipedia.org/wiki/Tikhonov_regularization $\endgroup$ – boscovich Oct 15 '14 at 13:57
  • $\begingroup$ Small point of clarification, these are slides from Ryan Tibshirani not Rob. $\endgroup$ – Ellis Valentiner Oct 15 '14 at 14:03
  • $\begingroup$ ok, thanks a lot for the clarification. But I don't understand why squared for L2 and not squared for L1. Don't we have a general formulae for any kind of regularization? $\endgroup$ – PLOTZ Oct 15 '14 at 14:05
  • $\begingroup$ @user12202013: thank you for pointing that out. I didn't notice that. $\endgroup$ – boscovich Oct 16 '14 at 11:16
9
$\begingroup$

Ridge and lasso are two ways of $\ell_p$ regularization:

$ \ell_1(x) = \sum_i (|x|^p)^{1/p} $

Lasso regression puts a penalty on the $\ell_1$ norm:

$ ||\beta||_1 = \ell_1(\beta) = \sum_i (|\beta_i|^1)^\frac{1}{1} = \sum_i \beta_i $

Ridge regression puts a penalty on the square of the $\ell_2$ norm:

$ ||\beta||_2^2 = \ell_2(\beta)^2 = \sum_i (|\beta_i|^2) $

As you can see the square-root from the $\ell_2$ norm is cancelled by squaring the norm. Without squaring the norm, the penalty would be the euclidean length of the coefficients.

The benefit of the squared $\ell_2$ norm is that it penalizes large coefficients more. Both the euclidean length and the square make the largest coefficient the most important one.

The benefit of the $\ell_1$ norm is that it penalizes every increase in coefficients equally. Hence, a zero coefficient is noticably better than a small coefficient. Thus the coefficient will become sparse.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ This is misleading. $\|\boldsymbol{x}\|_1 = \sum_i | x_i |$, there’s no squares or square roots in here. en.wikipedia.org/wiki/Norm_(mathematics) $\endgroup$ – Tim Jun 4 at 21:54
  • $\begingroup$ Can you clarify why it is misleading? I asume you mean that there is no square in the lasso equation you show. But there is, because $|x_i| = \sqrt{x_i^2}$. $\endgroup$ – Pieter Jun 6 at 7:50
  • $\begingroup$ You describe it as if it all was about combinations of squares and square roots, while the point is that $L_p$ norm is about $p$-th powers $(\sum_i |x|^p)^{1/p}$. $\endgroup$ – Tim Jun 6 at 9:28
  • $\begingroup$ I did a rewrite that shows the relation with the $\ell_p$ norm better $\endgroup$ – Pieter Jun 10 at 10:13
7
$\begingroup$

There are lots of penalized approaches that have all kinds of different penalty functions now (ridge, lasso, MCP, SCAD). The question of why is one of a particular form is basically "what advantages/disadvantages does such a penalty provide?".

Properties of interest might be:

1) nearly unbiased estimators (note all penalized estimators will be biased)

2) Sparsity (note ridge regression does not produce sparse results i.e. it does not shrink coefficients all the way to zero)

3) Continuity (to avoid instability in model prediction)

These are just a few properties one might be interested in a penalty function.

It is a lot easier to work with a sum in derivations and theoretical work: e.g. $||\beta||_2^2=\sum |\beta_i|^2$ and $||\beta||_1 = \sum |\beta_i|$. Imagine if we had $\sqrt{\left(\sum |\beta_i|^2\right)}$ or $\left( \sum |\beta_i|\right)^2$. Taking derivatives (which is necessary to show theoretical results like consistency, asymptotic normality etc) would be a pain with penalties like that.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ ok, thanks. But why squared for L2 and not squared for L1? Don't we have a general formulae for any kind of regularization? This is puzzling me... $\endgroup$ – PLOTZ Oct 15 '14 at 14:34
  • $\begingroup$ @PLOTZ I added a bit to my answer. $\endgroup$ – bdeonovic Oct 15 '14 at 17:19
  • $\begingroup$ Thanks a lot Benjamin! For sure it's clearer now! I didn't get this theoretical purpose before your answer. Many thanks for your answer. $\endgroup$ – PLOTZ Oct 16 '14 at 5:46
  • $\begingroup$ @Benjamin: in point #1 did you actually mean "(not all penalized estimators will be unbiased)"? Ridge regression –just to name one– is biased. $\endgroup$ – boscovich Oct 16 '14 at 7:40
  • $\begingroup$ whoops yes thanks for catching that! I think in fact all penalized estimators will be biased. $\endgroup$ – bdeonovic Oct 16 '14 at 11:37
5
$\begingroup$

Actually both the square of the $\ell_2$-norm and the $\ell_1$-norm come from a same class of regularization: $\|\boldsymbol{\beta}\|_p^p$ when $p > 0$.

The Ridge regression is then using $p=2$, and the Lasso $p=1$ but one can use other values of $p$.

For example you have sparse solution for all values of $p \leq 1$, and the smaller the value of $p$ the sparser the solution.

For values of $p \leq 1$ your objective is no more smooth so the optimization become harder; for $p<1$ the objective is non-convex and so the optimization even harder...

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

I believe there is an even simpler answer here, although "why" questions are always hard to answer when a technique is developed. The squared $l_2$-norm is used so that the regularization term is easily differentiable. Ridge regression minimizes:

$$\|\mathbf{y - X\beta}\|^2_2+\lambda\|\beta\|_2^2$$

Which can also be written: $$\|\mathbf{y - X\beta}\|^2_2+\lambda\beta^T\beta$$

This can now be easily differentiated wrt $\beta$ to get the closed-form solution:

$$\hat\beta^{\text{ridge}} = (\mathbf{X}^T\mathbf{X} + \lambda I)^{-1}\mathbf{X}^T\mathbf{y}$$

from which all kinda of inference can be derived.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Consider one other important difference between using the square of the $\ell_2$ norm (i.e. ridge regression) and the unmodified $\ell_2$ norm: the derivative of the $\ell_2$ norm of $x$, $||x||_2$, at $x$ is given by $\frac{x}{ ||x||_2}$ and therefore not differentiable at the zero vector. That is, although the $\ell_2$ norm does not do individual variable selection like the lasso, it could theoretically yield $\beta=0$ as the solution to the maximum penalized likelihood. By squaring the $\ell_2$ norm in the penalty, the ridge-type penalty is differentiable everywhere and can never yield such a solution.

This behavior is exactly (by my understanding) why the group lasso (Yuan and Lin) and the sparse group lasso (Simon, et al.), etc, use the $\ell_2$ norm (on prespecified subsets of the coefficients) instead of the square of the $\ell_2$ norm.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Another interpretation of it is the Bayesian one. In a frequentist approach the regularization with $$\|\cdot\|_2^2$$ is equivalent to having a Gaussian prior on your weight vector $\beta$. See here for example: Why is the L2 regularization equivalent to Gaussian prior?

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.