2
$\begingroup$

This question already has an answer here:

I'm learning about the Statistical learning and in the section comparing Lasso and Ridge Regression it shows that the main difference between these two problems is the way the constraint/penalty is formulated.

In Lasso, the penalty is $\ell_1$ norm: $\lambda \sum |\beta_j|$, while in regression, the penalty is $\ell_2$: $\lambda \sum \beta_j^2$.

Geometrically, this means that the lasso will have a constraint in the form of a diamond (in 2 dimensions), and in higher dimensions it will have vertices and edges. For ridge regression, in 2D, it is a circle, and hypersphere in higher dimensions.

My question is: The author claims that you get SPARSITY in the lasso. I do not understand why, even with the geometric picture above. And what is the clear advantage of Lasso over ridge regression?

Your insights would be very valuable. I appreciate if your answer would contain some mathematics, but more importantly, intuition. Thanks

$\endgroup$

marked as duplicate by Scortchi May 12 '15 at 15:26

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

5
$\begingroup$

The lasso penalty will force some of the coefficients quickly to zero. This means that variables are removed from the model, hence the sparsity.

Ridge regression will more or less compress the coefficients to become smaller. This does not necessarily result in 0 coefficients and removal of variables.

enter image description here

See the picture above, taken from onlinecourses.science.psu.edu/stat857/node/158

The circles represent the error function and the magenta area the possible values for the parameters.

You can see that in the left graph that the function is likely to hit the possible value space on one of the corners, on the axes. This implies that β1 is 0. On the right, where the space of allowed values is round due to the quadratic constraint, the function can hit the possible space in more arbitrary places

$\endgroup$
  • $\begingroup$ Thanks for your explanation. So this is where 'sparsity' comes from, coefficients turning ito zero. But can you please explain how lasso forces coefficients to zero, while ridge doesn't? $\endgroup$ – cgo May 12 '15 at 14:29
  • 2
    $\begingroup$ No trivial, but see onlinecourses.science.psu.edu/stat857/node/158 In the first picture the circles represent the error function and the magenta area the possible values for the parameters. You can see that in the left graph that the function is likely to hit the possible value space on one of the corners on the axes. This implies that $\beta1$ is $0$. On the right, where the space of allowed values is round due to the quadratic constraint, the function can hit the possible space in more arbitrary places. $\endgroup$ – spdrnl May 12 '15 at 14:39
  • 1
    $\begingroup$ Good comment, maybe you could include it in the answer? If you could include the picture, too, that would be even better. $\endgroup$ – Richard Hardy May 12 '15 at 14:52

Not the answer you're looking for? Browse other questions tagged or ask your own question.