Sparsity in Lasso and advantage over ridge (Statistical Learning) [duplicate]

I'm learning about the Statistical learning and in the section comparing Lasso and Ridge Regression it shows that the main difference between these two problems is the way the constraint/penalty is formulated.

In Lasso, the penalty is $\ell_1$ norm: $\lambda \sum |\beta_j|$, while in regression, the penalty is $\ell_2$: $\lambda \sum \beta_j^2$.

Geometrically, this means that the lasso will have a constraint in the form of a diamond (in 2 dimensions), and in higher dimensions it will have vertices and edges. For ridge regression, in 2D, it is a circle, and hypersphere in higher dimensions.

My question is: The author claims that you get SPARSITY in the lasso. I do not understand why, even with the geometric picture above. And what is the clear advantage of Lasso over ridge regression?

Your insights would be very valuable. I appreciate if your answer would contain some mathematics, but more importantly, intuition. Thanks

marked as duplicate by Scortchi♦May 12 '15 at 15:26

The lasso penalty will force some of the coefficients quickly to zero. This means that variables are removed from the model, hence the sparsity.

Ridge regression will more or less compress the coefficients to become smaller. This does not necessarily result in 0 coefficients and removal of variables. See the picture above, taken from onlinecourses.science.psu.edu/stat857/node/158

The circles represent the error function and the magenta area the possible values for the parameters.

You can see that in the left graph that the function is likely to hit the possible value space on one of the corners, on the axes. This implies that β1 is 0. On the right, where the space of allowed values is round due to the quadratic constraint, the function can hit the possible space in more arbitrary places

• Thanks for your explanation. So this is where 'sparsity' comes from, coefficients turning ito zero. But can you please explain how lasso forces coefficients to zero, while ridge doesn't? – cgo May 12 '15 at 14:29
• No trivial, but see onlinecourses.science.psu.edu/stat857/node/158 In the first picture the circles represent the error function and the magenta area the possible values for the parameters. You can see that in the left graph that the function is likely to hit the possible value space on one of the corners on the axes. This implies that $\beta1$ is $0$. On the right, where the space of allowed values is round due to the quadratic constraint, the function can hit the possible space in more arbitrary places. – spdrnl May 12 '15 at 14:39
• Good comment, maybe you could include it in the answer? If you could include the picture, too, that would be even better. – Richard Hardy May 12 '15 at 14:52