$L_1$ regularization LASSO regression. $L_2$ regularization is is Tikhonov regularization or ridge regression. The combination of the two is elastic net regularization.
To what do the $L_1$ and $L_2$ refer? What is the origin of this notation?
$\begingroup$
$\endgroup$
2
-
4$\begingroup$ You are mixing up $L_2$ is ridge regression and $L_1$ is Lasso. $\endgroup$– Chamberlain MbahCommented Mar 15, 2018 at 11:19
-
$\begingroup$ $L_1$ and $L_2$ are not even correct. Those refer to other norms. When authors use $L_1$ and $L_2$, they usually mean the $1$-norm (Manhattan distance) and the $2$-norm (Euclidean distance), respectively. $\endgroup$– Rodrigo de AzevedoCommented Mar 16, 2018 at 10:36
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
7
These regularization techniques are often encountered with variable selection. Assume you have a regression model $$ Y=\beta_0+\beta_1X_1+\ldots+\beta_NX_N+\epsilon. $$ Your intention is select a few predictors and hence use a prediction model with those predictors. You will often go for the $X_i$ with the largest $\beta_i$. It turns out that if you first select and try to estimate $\beta_i$, your estimator $\hat \beta_i$ is positively biased (i.e., larger on average than $\beta_i$). Regularization is used to reduce this positive bias. Say you were to estimate $\mathbf{\beta}=(\beta_0,\beta_1,\ldots,\beta_N)^T$ by least squares, then you will do something like $$ \hat \beta=\text{argmin}_\mathbf{\beta}\left\{(Y-\mathbf{X}\mathbf{\beta})^2\right\} $$ and return the largest $\hat \beta_i$s. Like I said these $\hat \beta_i$ s are positively biased. To counteract this bias you could regularize your least square action. Instead find $$ \hat \beta^\text{Ridge}=\text{argmin}_\mathbf{\beta}\left\{(Y-\mathbf{X}\mathbf{\beta})^2+\lambda \sum_{i=1}^{N}\beta_i^2\right\} $$ or
$$ \hat \beta^\text{Lasso}=\text{argmin}_\mathbf{\beta}\left\{(Y-\mathbf{X}\mathbf{\beta})^2+ \lambda\sum_{i=1}^{N}|\beta_i|\right\} $$ The ridge uses the $L_2$ regularization and the lasso uses the $L_1$ regularisation. In general we could refer to the type of regularizations as $L_p$ where an $L_p$ regularazation takes the form $$ \hat \beta^\text{general}=\text{argmin}_\mathbf{\beta}\left\{(Y-\mathbf{X}\mathbf{\beta})^2+\lambda\sum_{i=1}^{N}|\beta_i|^p.\right\} $$ So you see that when $p=1$ we get the lasso and $p=2$ the ridge and so on. Mathematicians general refer to the function say $L_p(\mathbf x)=\left(\sum_{i=1}^{N}|x_i|^p\right)^{1/p}$ as a norm placed on the vector $\mathbf x=(x_1,\ldots,x_N)^T$.
-
4$\begingroup$ (+1) You're missing tuning parameters here ($\lambda$). It may also be useful here to talk about $L_1$ and $L_2$ norms. $\endgroup$ Commented Mar 15, 2018 at 11:40
-
1$\begingroup$ Indeed let me add these. $\endgroup$ Commented Mar 15, 2018 at 11:42
-
1$\begingroup$ The ridge solution does not need a square root. Also note that the brackets after argmin should bracket all that follows, not just the sum of squared errors part. It would be great if you could edit the answer accordingly. Moreover, I would not start by addressing variable selection given that ridge does not do variable selection... And more importantly, does your answer address the real question? I am not entirely sure. (After the edit, the last bit is perhaps irrelevant.) $\endgroup$ Commented Mar 15, 2018 at 11:48
-
1$\begingroup$ Outside the summation in the general case the correct power is $1/p$ (well, when talking about $\ell_p$-norms, that is. People usually employ the squared $\ell_2$-norm instead for example). $\endgroup$– FirebugCommented Mar 15, 2018 at 12:02
-
1$\begingroup$ Once again, the ridge solution does not have a square root. $\endgroup$ Commented Mar 15, 2018 at 12:04
$\begingroup$
$\endgroup$
4
The "1" and "2" in $L_{1}$ and $L_{2}$ simply refers to the exponent you use in the regularization component of the function you minimize to find the function coefficients.
Thus, when using $L_{1}$ regularization, you're solving: $$ \min_{\beta} [[Y-X\beta]^2 + \lambda\sum_{i=1}^{n}|\beta_{i}|^{1}] $$
Whereas when using $L_{2}$ regularization, you're solving: $$ \min_{\beta}[[Y-X\beta]^2 + \lambda\sqrt{\sum_{i=1}^{n}|\beta_{i}|^{2}}] $$
(Note the exponent in the second term of the formula).
This is related to the concept of $L_{p}$-norm, defined as $\left ( \sum_{i=1}^{n}|x_{i}|^{p}\right )^{1/p}$
-
$\begingroup$ The ridge solution does not have a square root. $\endgroup$ Commented Mar 15, 2018 at 12:04
-
1$\begingroup$ what are you minimizing over? $\beta$, $X$ or $Y$. You do not have to assume people know what you are talking about. $\endgroup$ Commented Mar 15, 2018 at 12:13
-
$\begingroup$ @ChamberlainFoncha thank you for the comment, I've updated the solution now. $\endgroup$ Commented Mar 15, 2018 at 12:15
-
$\begingroup$ @RichardHardy I believe it's fine to have the square root there, and it matches the definition of a norm. You would just scale lambda and the solution should be equivalent. $\endgroup$ Commented Mar 15, 2018 at 12:17