Let's look at the image you used once again. Notice that it shows the same ovals of the cost function of linear regression $\hat \beta$ and the two different penalties: the lasso (square) and ridge regression (circle). We are interested in the first intersection point. With the ridge $\ell_1$ penalty, the closest intersection point is some point at the edge of the circle. With the lasso, it's the corner at zero.

Let's also look at one more example taken from here. You may notice that for different points it is the case that for lasso the intersection happens on the corners, while for ridge at different locations.

Next, let's go one step further and consider not only $\ell_2$ and $\ell_1$ penalties, but also $\ell_p$ with $0 < p < 1$. As you can see in the image below, it's even more "spiky" and has a concave shape. In such a case, the "spike" is something that the linear regression ovals will touch almost always because it's the most sticking out point. With $\ell_1$ it's less extreme, but the same logic follows.

In the end, let's go back to geometry. The circle is a geometric shape such that all the points on its boundary are equidistant from the center. In the case of squares, the corners are the only such points. If something approaches the shapes from the outside, it is likely to first touch the most external point, that would be the corners for the square and any point on the boundary for the circle.

Finally, this is visible even if we use your picture but for linear regression cost, we draw a perfect oval. Notice that it touches the non-zero point faster than the zero.

Let's look at the image you used once again. Notice that it shows the same ovals of the cost function of linear regression $\hat \beta$ and the two different penalties: the lasso (square) and ridge regression (circle). We are interested in the first intersection point. With the ridge $\ell_1$ penalty, the closest intersection point is some point at the edge of the circle. With the lasso, it's the corner at zero.

Let's also look at one more example taken from here. You may notice that for different points it is the case that for lasso the intersection often happens on the corners, while for ridge at different locations.

Next, let's go one step further and consider not only $\ell_2$ and $\ell_1$ penalties, but also $\ell_p$ with $0 < p < 1$. As you can see in the image below, it's even more "spiky" and has a concave shape. In such a case, the "spike" is something that the linear regression ovals will touch almost always because it's the most sticking out point. With $\ell_1$ it's less extreme, but the same logic applies.

In the end, let's go back to geometry. The circle is a geometric shape such that all the points on its boundary are equidistant from the center. In the case of squares, the corners are the most distant points from the center. If something approaches the shapes from the outside, it is likely to first touch the most external point, that would be the corners for the square and any point on the boundary for the circle.

Finally, this is visible even if we use your picture but for linear regression cost, we draw a perfect oval. Notice that it touches the non-zero point faster than the zero.

Let's look at the image you used once again. Notice that it shows the same ovals of the cost function of linear regression $\hat \beta$ and the two different penalties: the lasso (square) and ridge regression (circle). We are interested in the first intersection point. With the ridge $\ell_1$ penalty, the closest intersection point is some point at the edge of the circle. With the lasso, it's the corner at zero.

Let's also look at one more example taken from here. You may notice that for different points it is the case that for lasso the intersection happens on the corners, while for ridge at different locations.

Next, let's go one step further and consider not only $\ell_2$ and $\ell_1$ penalties, but also $\ell_p$ with $0 < p < 1$. As you can see in the image below, it's even more "spiky" and has a concave shape. In such a case, the "spike" is something that the linear regression ovals will touch almost always because it's the most sticking out point. With $\ell_1$ it's less extreme, but the same logic follows.

In the end, let's go back to geometry. The circle is a geometric shape such that all the points on its boundary are equidistant from the center. In the case of squares, the corners are the only such points. If something approaches the shapes from the outside, it is likely to first touch the most external point, that would be the corners for the square and any point on the boundary for the circle.

Let's look at the image you used once again. Notice that it shows the same ovals of the cost function of linear regression $\hat \beta$ and the two different penalties: the lasso (square) and ridge regression (circle). We are interested in the first intersection point. With the ridge $\ell_1$ penalty, the closest intersection point is some point at the edge of the circle. With the lasso, it's the corner at zero.

Let's also look at one more example taken from here. You may notice that for different points it is the case that for lasso the intersection happens on the corners, while for ridge at different locations.

Next, let's go one step further and consider not only $\ell_2$ and $\ell_1$ penalties, but also $\ell_p$ with $0 < p < 1$. As you can see in the image below, it's even more "spiky" and has a concave shape. In such a case, the "spike" is something that the linear regression ovals will touch almost always because it's the most sticking out point. With $\ell_1$ it's less extreme, but the same logic follows.

In the end, let's go back to geometry. The circle is a geometric shape such that all the points on its boundary are equidistant from the center. In the case of squares, the corners are the only such points. If something approaches the shapes from the outside, it is likely to first touch the most external point, that would be the corners for the square and any point on the boundary for the circle.

Finally, this is visible even if we use your picture but for linear regression cost, we draw a perfect oval. Notice that it touches the non-zero point faster than the zero.