8
$\begingroup$

Since this question was already ask here, I hope that it isn't a duplicate because it isn't answered. In my question I'll use another graph to make it more clear:

enter image description here

The left plot is describing the lasso-problem. My question is regarding to the cases when it is more likely to intersect the corner (in this case $\beta_j=0$) or the side of the constrained area.

I more or less figured out two dependencies:

  1. As in the picture, if we increase the budget of the constrained area or decrease the tuning parameter $\lambda$, the diamond will be getting bigger. It will be more likely to intersect with a side as with a Corner.

  2. The intersection depends on the "graphical position" of the OLS solution. But I can't explain to myself how the dependence works. Can some of you give me some graphical intuition in the case of $p=2$, when the LASSO will not set coefficients to zero?

$\endgroup$
5
$\begingroup$

Recall that the Lasso minimization problem can be viewed as the minimization of two terms: $OLS + L_1$. To answer your question:

The intersection depends on the "graphical position" of the OLS solution. But I can't explain to myself how the dependence works.

  • The solution to the constrained optimization lies at the intersection between the contours of the two functions, and this intersection varies as a function of $\lambda$. For $\lambda = 0$ the solution is the MLE (as usual) and for $\lambda = \infty$ the solution is at $[0,0]$.
  • Since at the vertices of the diamond, one or many of the variables have value 0, there is a non zero probability that one or many of the coefficients will have a value exactly equal to 0.

As you can see from your picture, there must be some values of $\lambda$ for which the solution does not take place at the vertex of the diamond. The solution cannot simply jump from the OLS minimum (when $\lambda = 0$) to a vertex of the diamond.

This is a misunderstanding many people have. The lasso doesn't magically set coefficients to zero ! It optimizes the lasso cost function, and the particular structure of this optimization problem makes it likely that the solution lies at the vertex of the diamond.

Can some of you give me some graphical intuition in the case of p=2, when the LASSO will not set coefficients to zero?

The answer is every time the optimal solutions is not at a vertex of the diamond. Now to answer the question you have not asked.

What makes it more likely for the lasso coefficient to not be zero

One obvious factor is when features are correlated - or if there is strong multi-correlation in your data set. Visually this will have the effect of "flattening" the OLS cost function in one direction (in 2d) and thus will strongly influence the path of the lasso solution by forcing it to take a less "direct" path towards one of the vertex. See this picture for a case where the two features are very correlated.

ridge_lasso_paths

If there were less correlation, the OLS contour plot would look more circular and the lasso solution may converge faster to a vertex. But again this depends on the particular shape of the OLS cost function for your given data set. In this case the OLS solution has a valley that is at an angle w.r.t the $0,0$ point. This where the notion of graphical position comes from

Sources

This post is strongly based on my previous post - For anyone interested, you can find most of the code and associated mathematical derivations on my blog and at this page

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Great post again ! It seems, that you have an answer to every question ;) There should be another dependence, or ? I mean depending on the OLS-Solution, there should be a different path, too. If we are looking at my picture above, assume that beta2 would be much smaller (something around 1). Then it is also more likely that we will hit a corner. What would be an intuition for that ? Depending on the size of beta2, beta 1 would be shrinked faster to zero $\endgroup$ – rook1996 Jun 18 '18 at 10:00
  • $\begingroup$ Possibly - but remember all of this is for normalized data - but yes the closer the OLS solution to one of the vertex, the faster the lasso will converge to a vertex. Remember that the OLS and Lasso contour plots are actually a valley and an inverted pyramid in 3D - so depending on the specifics of the OLS, the path of "least cost" may change $\endgroup$ – Xavier Bourret Sicotte Jun 18 '18 at 10:08
4
$\begingroup$

The ellipses are the equi-value contours of the loss, and the blue shapes are the regions of the penalty. By duality, the optimum occurs when a contour line intersects with a boundary of a blue shape.

The case you've shown is indeed one that foils Lasso sparsity - the axis of the ellipses is perpendicular to the surface of the diamond. This is possible, but not very likely. In higher dimensions, it is even less likely. For the 2D case it is easy to see that when the axis is not perpendicular to the surface of the diamond, the intersection will be either on the x or y axis.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for you answer. Could you try to connect your graphical explanation with some content about the lasso or extend your answer ? I can't still figure out in terms of an LASSO or OLS Interpretation, when the intersection wont be on an axis. There should be a dependence between the OLS solution (position of the Center of the ellipses) and the regarding LASSO solutions. In particular I want to gain some intuition about the two dependencies, that I listed above. The first one should be clear. $\endgroup$ – rook1996 Jun 6 '18 at 14:20
  • 1
    $\begingroup$ @rook1996 Sure, but I'll get to it in a few hours. $\endgroup$ – Ami Tavory Jun 6 '18 at 14:59
  • $\begingroup$ Is that question too hard or trivial ? $\endgroup$ – rook1996 Jun 14 '18 at 10:08
  • 1
    $\begingroup$ Too too busy :-) Sorry, I apologize - I'll try over the weekend. $\endgroup$ – Ami Tavory Jun 14 '18 at 13:59

Not the answer you're looking for? Browse other questions tagged or ask your own question.