14
$\begingroup$

Whenever someone writes about Lasso and Ridge Regression thy draw this diagram with the circle or with the diamond.

enter image description here

In the case of the diamond (Lasso regression) it is then always stated that Lasso forces one of the coefficients to 0. Therefor it introduces sparsity. I understand it somehow, but whenever I see the diagram my doubts return. Why couldn't one just draw it like this:

enter image description here

Obviously none of the coefficients is forced to zero in this case. Both can take number between -1 and 1. What am I missing? My drawing has to be wrong, but I don't get it why they always draw so that it hits $\beta_1=0$

Edit:

Just found this quote:

However, the lasso constraint has corners at each of the axes and so the ellipse will often intersect the constraint region at an axis

Is that it? It will intersect often with the constraint region, but it doesn't have to? Can't wrap my head around it. I can only imagine that in higher dimensional cases hitting a corner becomes more likely or even inevitable.

Improve this question
$\endgroup$
1
  • 5
    $\begingroup$ "Is that it? It will intersect often with the constraint region, but it doesn't have to? " yes, exactly. What you have drawn is not impossible, it is merely unlikely. $\endgroup$
    – littleO
    Commented Jun 25, 2019 at 4:07

2 Answers 2

Reset to default
12
$\begingroup$

Each circle around your point $\beta$ is actually an isoline in the 3rd dimension, i.e. upwards, and every points on such a line have the same value for the loss function. You could draw infinitely many such lines because these are visual simplification of something that should be a surface.

To answer your question: draw an additional isoline a bit further and you will get one that intersects with the vertices of your square.

It is not true that the lasso forces parameters to be zero immediately... what is true is that lasso leads parameters to converge to zero asymptotically as a function of $\alpha$ the lasso coefficient.

Here is a picture of the actual path of parameters on your such graph: taken from Lasso regression feature selection

enter image description here

And here is a different visualization taken from: Graphical path Coordinate Descent in case of semi-differentiable functions such as Lasso in 3D

enter image description here

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you, that was very informative. From the upper image I take that there is always an intersection between the surface and an edge of the L1 regularization. What I don't see yet is why this should be the point. Why should it end there? Isn't it in this example likely that the optimum of the loss function + penalty is at roughly [2.5, -5]? What forces the convergence towards a square vertices? $\endgroup$
    – Mr.Sh4nnon
    Commented Jun 24, 2019 at 17:07
  • 1
    $\begingroup$ Nothing forces anything - it is just the nature and structure of this particular optimization problem that leads to solutions which tend to be sparse - but i m sure someone could find a counter example or an edge case with a non sparse solution for some value of alpha - to understand this better you must think of it as an optimization problem $\endgroup$
    – Xavier Bourret Sicotte
    Commented Jun 24, 2019 at 19:52
3
$\begingroup$

They are not forced in either but the likleyhood of L1 intersecting with 0 is higher. This can be seen in the first image, for the same level curve, because of L2's circle popping out, it prevents clear access to the axis while L1's axis is less obscured by its diamond. Thus L1 is more likely to yield coefficents with 0.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the quick respond. That covers what I have added in the edit. So I guess my assumption is correct, that in higher dimensions hitting a corner becomes even more likely. But it doesn't have to right? $\endgroup$
    – Mr.Sh4nnon
    Commented Jun 24, 2019 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.