The graphical intuiton of the LASSO in case p=2 [duplicate]

Question

Since this question was already ask here, I hope that it isn't a duplicate because it isn't answered. In my question I'll use another graph to make it more clear:

The left plot is describing the lasso-problem. My question is regarding to the cases when it is more likely to intersect the corner (in this case $\beta_j=0$) or the side of the constrained area.

I more or less figured out two dependencies:

1. As in the picture, if we increase the budget of the constrained area or decrease the tuning parameter $\lambda$, the diamond will be getting bigger. It will be more likely to intersect with a side as with a Corner.

2. The intersection depends on the "graphical position" of the OLS solution. But I can't explain to myself how the dependence works. Can some of you give me some graphical intuition in the case of $p=2$, when the LASSO will not set coefficients to zero?

Answer 1

Recall that the Lasso minimization problem can be viewed as the minimization of two terms: $OLS + L_1$. To answer your question:

The intersection depends on the "graphical position" of the OLS solution. But I can't explain to myself how the dependence works.

• The solution to the constrained optimization lies at the intersection between the contours of the two functions, and this intersection varies as a function of $\lambda$. For $\lambda = 0$ the solution is the MLE (as usual) and for $\lambda = \infty$ the solution is at $[0,0]$.
• Since at the vertices of the diamond, one or many of the variables have value 0, there is a non zero probability that one or many of the coefficients will have a value exactly equal to 0.

As you can see from your picture, there must be some values of $\lambda$ for which the solution does not take place at the vertex of the diamond. The solution cannot simply jump from the OLS minimum (when $\lambda = 0$) to a vertex of the diamond.

This is a misunderstanding many people have. The lasso doesn't magically set coefficients to zero ! It optimizes the lasso cost function, and the particular structure of this optimization problem makes it likely that the solution lies at the vertex of the diamond.

Can some of you give me some graphical intuition in the case of p=2, when the LASSO will not set coefficients to zero?

The answer is every time the optimal solutions is not at a vertex of the diamond. Now to answer the question you have not asked.

What makes it more likely for the lasso coefficient to not be zero

One obvious factor is when features are correlated - or if there is strong multi-correlation in your data set. Visually this will have the effect of "flattening" the OLS cost function in one direction (in 2d) and thus will strongly influence the path of the lasso solution by forcing it to take a less "direct" path towards one of the vertex. See this picture for a case where the two features are very correlated.

If there were less correlation, the OLS contour plot would look more circular and the lasso solution may converge faster to a vertex. But again this depends on the particular shape of the OLS cost function for your given data set. In this case the OLS solution has a valley that is at an angle w.r.t the $0,0$ point. This where the notion of graphical position comes from

Sources

This post is strongly based on my previous post - For anyone interested, you can find most of the code and associated mathematical derivations on my blog and at this page

Answer 2

The ellipses are the equi-value contours of the loss, and the blue shapes are the regions of the penalty. By duality, the optimum occurs when a contour line intersects with a boundary of a blue shape.

The case you've shown is indeed one that foils Lasso sparsity - the axis of the ellipses is perpendicular to the surface of the diamond. This is possible, but not very likely. In higher dimensions, it is even less likely. For the 2D case it is easy to see that when the axis is not perpendicular to the surface of the diamond, the intersection will be either on the x or y axis.