8
$\begingroup$

I've a question regarding to the graphical intuition of the LASSO. I'm understanding why the lasso produce zero coefficient in case of intersecting a corner of the diamond. But I don't understand the case when the lasso Regression just shrink the paramters and don't set them to Zero. So if the RSS lines intersect with the "side" of the diamond. Assume in the case of the figure, that $\beta_1$ bigger and $\beta_2$ smaller. So the RSS-line would likely intersect a side.

Could you give me some intuition when the RSS-Line tangents with the side of the diamond instead of the corner?

enter image description here

$\endgroup$

2 Answers 2

12
$\begingroup$

Recall that the Lasso minimization problem can be expressed as:

$$ \hat \theta_{lasso} = argmin_{\theta \in \mathbb{R}^n} \sum_{i=1}^m (y_i - \mathbf{x_i}^T \theta)^2 + \lambda \sum_{j=1}^n | \theta_j| \ $$

Which can be viewed as the minimization of two terms: $OLS + L_1$.

  • The first OLS term can be written as $(y - X \theta)^T(y - X \theta) $ which gives rise to an elipse contour plot centered around the Maximum Likelihood Estimator.
  • The second $L_1$ term is the equation of a diamond centered around 0 (or a romboid in higher dimensions)
  • The solution to the constrained optimization lies at the intersection between the contours of the two functions, and this intersection varies as a function of $\lambda$. For $\lambda = 0$ the solution is the MLE (as usual) and for $\lambda = \infty$ the solution is at $[0,0]$.
  • Since at the vertices of the diamond, one or many of the variables have value 0, there is a non zero probability that one or many of the coefficients will have a value exactly equal to 0.

This last bullet is important in answering your question:

But I don't understand the case when the lasso Regression just shrink the paramters and don't set them to Zero

The lasso regression does't have to set coefficients to zero, in many cases it doesn't. What happens is that as you increase the $\lambda$ parameter, the probability that the solution takes place at a vertex of the diamond increases, and so the probability that one or many coefficients is exactly zero also increases.

Could you give me some intuition when the RSS-Line tangents with the side of the diamond instead of the corner?

Here is a graph I have produced based on simulated data. It shows the optimal solution for ridge and lasso regression as a function of the $\lambda$ parameter (lasso is on the right hand side).

ridge_lasso_paths

You can see that there are many solutions that are not on the vertex of the diamond!

The impact of strongly correlated features

This simple example shows what happens when the two features are highly correlated, in fact here $x_1 = x$ and $x_2 = x^2$ so they are so strongly correlated that the shape of the OLS cost function looks like an upside-down ridge, or a valley - hence the intuition behind the name ridge regression.

Sources

This post is strongly based on my previous post - For anyone interested, you can find most of the code and associated mathematical derivations on my blog and at this page

$\endgroup$
3
  • $\begingroup$ In the last bullet, you write and I quote "one or many of the variables have value 0, there is a non zero probability that one or many of the coefficients will have a value exactly equal to 0" . My confusion is, by variable do you mean parameter because while optimising/learning we are in parameter space. Typically in literature "variable" is synonym to "feature" and " parameters" is synonym to "coefficients". Is it used interchangeably. $\endgroup$
    – v09
    Mar 2, 2020 at 10:48
  • $\begingroup$ Note: In the linked sources(previous post). there is a similar bullet point where "coefficient" is replaced with "feature". I quote "one or many of the variables have value 0, there is a probability that one or many of the features will have a value exactly equal to 0." $\endgroup$
    – v09
    Mar 2, 2020 at 10:57
  • $\begingroup$ coefficient means theta while variables mean x - perhaps my wording confuses the two in the post but the iniition is that $\endgroup$ Mar 6, 2020 at 1:26
1
$\begingroup$

The role of the diamond is pretty clear, I guess: in brief, the region where the the (two in this picture) coefficients are such that the sum of their absolute values does not exceed the "budget" $s$.

The ellipses denote regions where the sum of squared residuals take identical values. The OLS estimate $\hat\beta$ is, by its very definition, the point where that sum is smallest. Hence, the further we move away from that point, the larger the sum gets.

We now seek a point that is such that it touches the region where the budget constraint is met while at the same time making the sum of squared residuals no larger than necessary.

$\endgroup$
2
  • $\begingroup$ That would be not the answer to my question, what are the dependent factors that the RSS-lines intersect a corner or a side $\endgroup$
    – Leo96
    May 31, 2018 at 19:04
  • $\begingroup$ your question comes from a miss-understanding of the problem - see my answer $\endgroup$ Jun 18, 2018 at 8:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.