Graphical interpretation of LASSO

Question

I've a question regarding to the graphical intuition of the LASSO. I'm understanding why the lasso produce zero coefficient in case of intersecting a corner of the diamond. But I don't understand the case when the lasso Regression just shrink the paramters and don't set them to Zero. So if the RSS lines intersect with the "side" of the diamond. Assume in the case of the figure, that $\beta_1$ bigger and $\beta_2$ smaller. So the RSS-line would likely intersect a side.

Could you give me some intuition when the RSS-Line tangents with the side of the diamond instead of the corner?

Answer 1

Recall that the Lasso minimization problem can be expressed as:

$$ \hat \theta_{lasso} = argmin_{\theta \in \mathbb{R}^n} \sum_{i=1}^m (y_i - \mathbf{x_i}^T \theta)^2 + \lambda \sum_{j=1}^n | \theta_j| \ $$

Which can be viewed as the minimization of two terms: $OLS + L_1$.

• The first OLS term can be written as $(y - X \theta)^T(y - X \theta) $ which gives rise to an elipse contour plot centered around the Maximum Likelihood Estimator.
• The second $L_1$ term is the equation of a diamond centered around 0 (or a romboid in higher dimensions)
• The solution to the constrained optimization lies at the intersection between the contours of the two functions, and this intersection varies as a function of $\lambda$. For $\lambda = 0$ the solution is the MLE (as usual) and for $\lambda = \infty$ the solution is at $[0,0]$.
• Since at the vertices of the diamond, one or many of the variables have value 0, there is a non zero probability that one or many of the coefficients will have a value exactly equal to 0.

This last bullet is important in answering your question:

But I don't understand the case when the lasso Regression just shrink the paramters and don't set them to Zero

The lasso regression does't have to set coefficients to zero, in many cases it doesn't. What happens is that as you increase the $\lambda$ parameter, the probability that the solution takes place at a vertex of the diamond increases, and so the probability that one or many coefficients is exactly zero also increases.

Could you give me some intuition when the RSS-Line tangents with the side of the diamond instead of the corner?

Here is a graph I have produced based on simulated data. It shows the optimal solution for ridge and lasso regression as a function of the $\lambda$ parameter (lasso is on the right hand side).

You can see that there are many solutions that are not on the vertex of the diamond!

The impact of strongly correlated features

This simple example shows what happens when the two features are highly correlated, in fact here $x_1 = x$ and $x_2 = x^2$ so they are so strongly correlated that the shape of the OLS cost function looks like an upside-down ridge, or a valley - hence the intuition behind the name ridge regression.

Sources

This post is strongly based on my previous post - For anyone interested, you can find most of the code and associated mathematical derivations on my blog and at this page

Answer 2

The role of the diamond is pretty clear, I guess: in brief, the region where the the (two in this picture) coefficients are such that the sum of their absolute values does not exceed the "budget" $s$.

The ellipses denote regions where the sum of squared residuals take identical values. The OLS estimate $\hat\beta$ is, by its very definition, the point where that sum is smallest. Hence, the further we move away from that point, the larger the sum gets.

We now seek a point that is such that it touches the region where the budget constraint is met while at the same time making the sum of squared residuals no larger than necessary.