1
$\begingroup$

$L_1$ regularization LASSO regression. $L_2$ regularization is is Tikhonov regularization or ridge regression. The combination of the two is elastic net regularization.

To what do the $L_1$ and $L_2$ refer? What is the origin of this notation?

$\endgroup$
  • 4
    $\begingroup$ You are mixing up $L_2$ is ridge regression and $L_1$ is Lasso. $\endgroup$ – Chamberlain Foncha Mar 15 '18 at 11:19
  • $\begingroup$ $L_1$ and $L_2$ are not even correct. Those refer to other norms. When authors use $L_1$ and $L_2$, they usually mean the $1$-norm (Manhattan distance) and the $2$-norm (Euclidean distance), respectively. $\endgroup$ – Rodrigo de Azevedo Mar 16 '18 at 10:36
5
$\begingroup$

These regularization techniques are often encountered with variable selection. Assume you have a regression model $$ Y=\beta_0+\beta_1X_1+\ldots+\beta_NX_N+\epsilon. $$ Your intention is select a few predictors and hence use a prediction model with those predictors. You will often go for the $X_i$ with the largest $\beta_i$. It turns out that if you first select and try to estimate $\beta_i$, your estimator $\hat \beta_i$ is positively biased (i.e., larger on average than $\beta_i$). Regularization is used to reduce this positive bias. Say you were to estimate $\mathbf{\beta}=(\beta_0,\beta_1,\ldots,\beta_N)^T$ by least squares, then you will do something like $$ \hat \beta=\text{argmin}_\mathbf{\beta}\left\{(Y-\mathbf{X}\mathbf{\beta})^2\right\} $$ and return the largest $\hat \beta_i$s. Like I said these $\hat \beta_i$ s are positively biased. To counteract this bias you could regularize your least square action. Instead find $$ \hat \beta^\text{Ridge}=\text{argmin}_\mathbf{\beta}\left\{(Y-\mathbf{X}\mathbf{\beta})^2+\lambda \sum_{i=1}^{N}\beta_i^2\right\} $$ or

$$ \hat \beta^\text{Lasso}=\text{argmin}_\mathbf{\beta}\left\{(Y-\mathbf{X}\mathbf{\beta})^2+ \lambda\sum_{i=1}^{N}|\beta_i|\right\} $$ The ridge uses the $L_2$ regularization and the lasso uses the $L_1$ regularisation. In general we could refer to the type of regularizations as $L_p$ where an $L_p$ regularazation takes the form $$ \hat \beta^\text{general}=\text{argmin}_\mathbf{\beta}\left\{(Y-\mathbf{X}\mathbf{\beta})^2+\lambda\sum_{i=1}^{N}|\beta_i|^p.\right\} $$ So you see that when $p=1$ we get the lasso and $p=2$ the ridge and so on. Mathematicians general refer to the function say $L_p(\mathbf x)=\left(\sum_{i=1}^{N}|x_i|^p\right)^{1/p}$ as a norm placed on the vector $\mathbf x=(x_1,\ldots,x_N)^T$.

$\endgroup$
  • 4
    $\begingroup$ (+1) You're missing tuning parameters here ($\lambda$). It may also be useful here to talk about $L_1$ and $L_2$ norms. $\endgroup$ – Greenparker Mar 15 '18 at 11:40
  • 1
    $\begingroup$ Indeed let me add these. $\endgroup$ – Chamberlain Foncha Mar 15 '18 at 11:42
  • 1
    $\begingroup$ The ridge solution does not need a square root. Also note that the brackets after argmin should bracket all that follows, not just the sum of squared errors part. It would be great if you could edit the answer accordingly. Moreover, I would not start by addressing variable selection given that ridge does not do variable selection... And more importantly, does your answer address the real question? I am not entirely sure. (After the edit, the last bit is perhaps irrelevant.) $\endgroup$ – Richard Hardy Mar 15 '18 at 11:48
  • 1
    $\begingroup$ Outside the summation in the general case the correct power is $1/p$ (well, when talking about $\ell_p$-norms, that is. People usually employ the squared $\ell_2$-norm instead for example). $\endgroup$ – Firebug Mar 15 '18 at 12:02
  • 1
    $\begingroup$ Once again, the ridge solution does not have a square root. $\endgroup$ – Richard Hardy Mar 15 '18 at 12:04
1
$\begingroup$

The "1" and "2" in $L_{1}$ and $L_{2}$ simply refers to the exponent you use in the regularization component of the function you minimize to find the function coefficients.

Thus, when using $L_{1}$ regularization, you're solving: $$ \min_{\beta} [[Y-X\beta]^2 + \lambda\sum_{i=1}^{n}|\beta_{i}|^{1}] $$

Whereas when using $L_{2}$ regularization, you're solving: $$ \min_{\beta}[[Y-X\beta]^2 + \lambda\sqrt{\sum_{i=1}^{n}|\beta_{i}|^{2}}] $$

(Note the exponent in the second term of the formula).

This is related to the concept of $L_{p}$-norm, defined as $\left ( \sum_{i=1}^{n}|x_{i}|^{p}\right )^{1/p}$

$\endgroup$
  • $\begingroup$ The ridge solution does not have a square root. $\endgroup$ – Richard Hardy Mar 15 '18 at 12:04
  • 1
    $\begingroup$ what are you minimizing over? $\beta$, $X$ or $Y$. You do not have to assume people know what you are talking about. $\endgroup$ – Chamberlain Foncha Mar 15 '18 at 12:13
  • $\begingroup$ @ChamberlainFoncha thank you for the comment, I've updated the solution now. $\endgroup$ – user3208442 Mar 15 '18 at 12:15
  • $\begingroup$ @RichardHardy I believe it's fine to have the square root there, and it matches the definition of a norm. You would just scale lambda and the solution should be equivalent. $\endgroup$ – user3208442 Mar 15 '18 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.