Parameters go to zero for lasso regularization function

Question

From the book Pattern Recognition and Machine Learning by Bishop I saw this picture which shows the contours of a regularization function. The advantage of this regularization function (right graph) is supposed to be that one of the parameters equals zero at some optimal solution(in this case w2 is zero at w*). I can see from the picture why this is true in this case, what I do not get is wy the the contours of the unregularized error function (blue lines) can not shift downwards such that the two shapes are tangent to each other on the side of the red contour such that none of the two parameters is equal to zero.

Answer 1

What you suggest is theoretically possible, but would require that $w_1 = w_2$ exactly and the probability of that happening is either 0, or close enough to 0 with real data that we really don't need to worry about it.

edit

Actually there is the other condition where this will be true, if the point in the middle of the contours moves down (without moving left or right) until it is below the top of the shaded area, then there will be a minimum in the constrained region with both values not equal to 0. Or if $w_1$ and $w_2$ are close enough to each other relative to the constraint/penalty value then there will be a value within the constrained region that does not force a coefficient to 0.

When doing a lasso analysis with very little constraint/penalty the coefficients will not be 0. But we usually use the Lasso because we want enough of a constraint to force some of the coefficients to 0.