I got a more theoretical question here: I have done some research about the L2 (Ridge) and L1 (Lasso) regularizations. I know the formula, and understand the aim of those two different procedures. The crucial difference between the two approaches is that L1 takes the absolute value while L2 takes the squared value of coefficients multiplied by the hyperparameter lambda. However, it is still not clear to me why L2 regularization cannot shrink parameters down to zero and L1 can. I can't see this from the formula. Could someone please explain this to me (probably with one example).

L1 Regularization:

enter image description here

L2 Regularization:

enter image description here

Thanks for your help!


Let's examine the coefficient estimates of ridge as a function of the OLS estimates.

The matrix expression which minimize the loss is

$$ \hat{\beta}^{\text {ridge }}=\left(\mathbf{X}^{T} \mathbf{X}+\lambda \mathbf{I}\right)^{-1} \mathbf{X}^{T} \mathbf{y}$$

According to the authors of Elements of Statistical Learning, each element of that vector can be written as

$$ \hat{\beta}_j/(1+\lambda)$$

It is easy to see why this coefficient can never be 0. Compare this to the Lasso, in which the coefficients are (in the case of orthonormal columns of the design matrix)

$$ \operatorname{sign}\left(\hat{\beta}_{j}\right)\left(\left|\hat{\beta}_{j}\right|-\lambda\right)_{+}$$

Here, $(x)_{+}$ denotes the positive part of x (so $x$ if $x>0$ and 0 otherwise). It is clear to see why (again, in the orthonormal case) why the LASSO can estimate some coefficients to be 0.

That's just me reading from ESL. Here is a more complete argument.


It's because of the difference between the gradients of the two regularizations.

For the L1 regularization $|w|$ (derived from the Laplace distribution), the gradient remains 1(as depicted on the right-hand side of the below picture) no matter how small the $|w|$ is, leading the $w$ quickly to shrink to 0. enter image description here

But for the L2 regularization $w^2$, the closer $w$ approaches 0(the regularization's smallest value) the smaller(approaching 0) its gradient $2w$ and the harder for it to be 0:

enter image description here


Not the answer you're looking for? Browse other questions tagged or ask your own question.