3
$\begingroup$

To explain the difference between Ridge and Lasso regression, following diagram is used as it is claimed that Ridge regression cannot shrink the regression coefficients to 0:

enter image description here

But my question is, if my equation of cost function is like the one on in the following image, Ridge can also reduce the coefficient to 0:

enter image description here

Please correct if I am wrong.

EDITS:

According to this answer(https://stats.stackexchange.com/a/30530/35989)

And now follows an interesting thing that greatly explains to me the difference between ridge regression and LASSO: in case of LASSO two contour plots will probably meet where the corner of regularizer is (𝛽1=0 or 𝛽2=0). In case of ridge regression that is almost never the case.

But it does not explain, why it can "ALMOST NEVER" be the case for ridge regression.

$\endgroup$
2
  • 6
    $\begingroup$ As you could learn from my answer to your other question nobody claims that the parameter won't be zero. It is just not the point of this regularization. The same, you could have parameters equal to zeros without using any regularization. You'll find good explanation of the geometry here stats.stackexchange.com/a/30530/35989 $\endgroup$
    – Tim
    Sep 1, 2022 at 18:49
  • $\begingroup$ Edited the question. Now I request the people to focus on the problem and not on the other unnecessary stuff $\endgroup$ Sep 2, 2022 at 5:37

2 Answers 2

4
$\begingroup$

Let's look at the image you used once again. Notice that it shows the same ovals of the cost function of linear regression $\hat \beta$ and the two different penalties: the lasso (square) and ridge regression (circle). We are interested in the first intersection point. With the ridge $\ell_1$ penalty, the closest intersection point is some point at the edge of the circle. With the lasso, it's the corner at zero.

enter image description here

Let's also look at one more example taken from here. You may notice that for different points it is the case that for lasso the intersection often happens on the corners, while for ridge at different locations.

enter image description here

Next, let's go one step further and consider not only $\ell_2$ and $\ell_1$ penalties, but also $\ell_p$ with $0 < p < 1$. As you can see in the image below, it's even more "spiky" and has a concave shape. In such a case, the "spike" is something that the linear regression ovals will touch almost always because it's the most sticking out point. With $\ell_1$ it's less extreme, but the same logic applies.

enter image description here

In the end, let's go back to geometry. The circle is a geometric shape such that all the points on its boundary are equidistant from the center. In the case of squares, the corners are the most distant points from the center. If something approaches the shapes from the outside, it is likely to first touch the most external point, that would be the corners for the square and any point on the boundary for the circle.

Finally, this is visible even if we use your picture but for linear regression cost, we draw a perfect oval. Notice that it touches the non-zero point faster than the zero.

enter image description here

$\endgroup$
3
$\begingroup$

It is not shrinking to zero

When estimated coefficients are zero in Ridge regression, then it is only because a parameter value crossed the value of 0. It is not because it shrank to zero, when the penalty is further increased the coefficient won't remain zero.

Below is an example image for showing that the difference between Lasso (shrinking to zero) and Ridge regression (crossing zero).

comparison

In the Lasso regression on the left you see that, as we increase the level of regularisation (reduce the $\ell_1$ norm) more and more coefficients will be zero and they remain zero when we increase the level of regularisation further.

In the Ridge regression on the right, the coefficient related to the green curve is shortly equal to zero when the $\ell_1$ norm of the coefficients is around 0.55. But the coefficient only passes zero and for stronger regularisation levels it is non-zero again.

The reason that coefficients can be zero with ridge regression is because the ridge regression follows a curved path. See Intuition for nonmonotonicity of coefficient paths in ridge regression

The probability of a zero coefficient

The above is more like a semantic comment about the terminology and meaning of 'shrinking'. If we ignore that, then you might still wonder why a zero coefficient is almost surely not occuring (it can occur but the probability is just zero).

This is similar to your other question Why in the Ridge regression, the coefficients cannot be 0? and is similar to $P[X=x]=0$ when $X$ is a continuous variable But here you are looking for a geometric interpretation, based on your drawing.

If we look at the case of Lasso, then we see that there are many circles that touch the square in the points/corners where one of the coefficients is zero. There is a range values for which this occurs and that range has a non-zero probability.

In the case of ridge there are only four possible single points such that the circles touch a circle in a points/corners where one of the coefficients is zero. For these single points there is zero probability that you have these as the OLS solution.

circles around circle or square

$\endgroup$
1
  • $\begingroup$ Your first 2 sentences are very insightful. $\endgroup$ Jun 23, 2023 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.