6
$\begingroup$

I'm learning about the Statistical learning and in the section comparing Lasso and Ridge Regression it shows that the main difference between these two problems is the way the constraint/penalty is formulated.

In Lasso, the penalty is $\ell_1$ norm: $\lambda \sum |\beta_j|$, while in regression, the penalty is $\ell_2$: $\lambda \sum \beta_j^2$.

Geometrically, this means that the lasso will have a constraint in the form of a diamond (in 2 dimensions), and in higher dimensions it will have vertices and edges. For ridge regression, in 2D, it is a circle, and hypersphere in higher dimensions.

My question is: The author claims that you get SPARSITY in the lasso. I do not understand why, even with the geometric picture above. And what is the clear advantage of Lasso over ridge regression?

Your insights would be very valuable. I appreciate if your answer would contain some mathematics, but more importantly, intuition. Thanks

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
12
$\begingroup$

The lasso penalty will force some of the coefficients quickly to zero. This means that variables are removed from the model, hence the sparsity.

Ridge regression will more or less compress the coefficients to become smaller. This does not necessarily result in 0 coefficients and removal of variables.

enter image description here

See the picture above, taken from onlinecourses.science.psu.edu/stat857/node/158

The circles represent the error function and the magenta area the possible values for the parameters.

You can see that in the left graph that the function is likely to hit the possible value space on one of the corners, on the axes. This implies that β1 is 0. On the right, where the space of allowed values is round due to the quadratic constraint, the function can hit the possible space in more arbitrary places

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for your explanation. So this is where 'sparsity' comes from, coefficients turning ito zero. But can you please explain how lasso forces coefficients to zero, while ridge doesn't? $\endgroup$
    – cgo
    May 12, 2015 at 14:29
  • 2
    $\begingroup$ No trivial, but see onlinecourses.science.psu.edu/stat857/node/158 In the first picture the circles represent the error function and the magenta area the possible values for the parameters. You can see that in the left graph that the function is likely to hit the possible value space on one of the corners on the axes. This implies that $\beta1$ is $0$. On the right, where the space of allowed values is round due to the quadratic constraint, the function can hit the possible space in more arbitrary places. $\endgroup$
    – spdrnl
    May 12, 2015 at 14:39
  • 1
    $\begingroup$ Good comment, maybe you could include it in the answer? If you could include the picture, too, that would be even better. $\endgroup$
    – Richard Hardy
    May 12, 2015 at 14:52
  • 1
    $\begingroup$ Does it possible for the contours of the error function hit the edge of the diamond? if so, what does it mean? $\endgroup$
    – FantasticAI
    Sep 15, 2020 at 0:53
  • $\begingroup$ @FantasticAI It is possible, and that means neither $\beta$'s are 0. However, also take note that hitting areas where neither $\beta$'s are zero is easier for ridge (edge of circle) than for lasso (edge of diamond, as you suppose). $\endgroup$
    – Nuclear241
    Nov 16, 2021 at 9:53

Not the answer you're looking for? Browse other questions tagged or ask your own question.