Geometrical interpretation of why can't ridge regression shrink coefficients to 0?

Question

To explain the difference between Ridge and Lasso regression, following diagram is used as it is claimed that Ridge regression cannot shrink the regression coefficients to 0:

But my question is, if my equation of cost function is like the one on in the following image, Ridge can also reduce the coefficient to 0:

Please correct if I am wrong.

EDITS:

According to this answer(https://stats.stackexchange.com/a/30530/35989)

And now follows an interesting thing that greatly explains to me the difference between ridge regression and LASSO: in case of LASSO two contour plots will probably meet where the corner of regularizer is (𝛽1=0 or 𝛽2=0). In case of ridge regression that is almost never the case.

But it does not explain, why it can "ALMOST NEVER" be the case for ridge regression.

Answer 1

Let's look at the image you used once again. Notice that it shows the same ovals of the cost function of linear regression $\hat \beta$ and the two different penalties: the lasso (square) and ridge regression (circle). We are interested in the first intersection point. With the ridge $\ell_1$ penalty, the closest intersection point is some point at the edge of the circle. With the lasso, it's the corner at zero.

Let's also look at one more example taken from here. You may notice that for different points it is the case that for lasso the intersection often happens on the corners, while for ridge at different locations.

Next, let's go one step further and consider not only $\ell_2$ and $\ell_1$ penalties, but also $\ell_p$ with $0 < p < 1$. As you can see in the image below, it's even more "spiky" and has a concave shape. In such a case, the "spike" is something that the linear regression ovals will touch almost always because it's the most sticking out point. With $\ell_1$ it's less extreme, but the same logic applies.

In the end, let's go back to geometry. The circle is a geometric shape such that all the points on its boundary are equidistant from the center. In the case of squares, the corners are the most distant points from the center. If something approaches the shapes from the outside, it is likely to first touch the most external point, that would be the corners for the square and any point on the boundary for the circle.

Finally, this is visible even if we use your picture but for linear regression cost, we draw a perfect oval. Notice that it touches the non-zero point faster than the zero.

Answer 2

It is not shrinking to zero

When estimated coefficients are zero in Ridge regression, then it is only because a parameter value crossed the value of 0. It is not because it shrank to zero, when the penalty is further increased the coefficient won't remain zero.

Below is an example image for showing that the difference between Lasso (shrinking to zero) and Ridge regression (crossing zero).

In the Lasso regression on the left you see that, as we increase the level of regularisation (reduce the $\ell_1$ norm) more and more coefficients will be zero and they remain zero when we increase the level of regularisation further.

In the Ridge regression on the right, the coefficient related to the green curve is shortly equal to zero when the $\ell_1$ norm of the coefficients is around 0.55. But the coefficient only passes zero and for stronger regularisation levels it is non-zero again.

The reason that coefficients can be zero with ridge regression is because the ridge regression follows a curved path. See Intuition for nonmonotonicity of coefficient paths in ridge regression

The probability of a zero coefficient

The above is more like a semantic comment about the terminology and meaning of 'shrinking'. If we ignore that, then you might still wonder why a zero coefficient is almost surely not occuring (it can occur but the probability is just zero).

This is similar to your other question Why in the Ridge regression, the coefficients cannot be 0? and is similar to $P[X=x]=0$ when $X$ is a continuous variable But here you are looking for a geometric interpretation, based on your drawing.

If we look at the case of Lasso, then we see that there are many circles that touch the square in the points/corners where one of the coefficients is zero. There is a range values for which this occurs and that range has a non-zero probability.

In the case of ridge there are only four possible single points such that the circles touch a circle in a points/corners where one of the coefficients is zero. For these single points there is zero probability that you have these as the OLS solution.