From the book Pattern Recognition and Machine Learning by Bishop I saw this picture which shows the contours of a regularization function. The advantage of this regularization function (right graph) is supposed to be that one of the parameters equals zero at some optimal solution(in this case w2 is zero at w*). I can see from the picture why this is true in this case, what I do not get is wy the the contours of the unregularized error function (blue lines) can not shift downwards such that the two shapes are tangent to each other on the side of the red contour such that none of the two parameters is equal to zero.enter image description here


1 Answer 1


What you suggest is theoretically possible, but would require that $w_1 = w_2$ exactly and the probability of that happening is either 0, or close enough to 0 with real data that we really don't need to worry about it.


Actually there is the other condition where this will be true, if the point in the middle of the contours moves down (without moving left or right) until it is below the top of the shaded area, then there will be a minimum in the constrained region with both values not equal to 0. Or if $w_1$ and $w_2$ are close enough to each other relative to the constraint/penalty value then there will be a value within the constrained region that does not force a coefficient to 0.

When doing a lasso analysis with very little constraint/penalty the coefficients will not be 0. But we usually use the Lasso because we want enough of a constraint to force some of the coefficients to 0.

  • $\begingroup$ I see your point however, if it is just a small shift such that the new value w** is equal to (w2* - epsilon, w1* +epsilon) then at that point the two parameter values would not be equal to each other but none of them would be zero correct? $\endgroup$
    – Andrew
    Commented Nov 23, 2015 at 19:02
  • 1
    $\begingroup$ @Andrew, thinking it through, I believe that you are correct, so there would actually be a triangle region (or possibly another shape) that would result in non-zero coefficients, and so the probability of being in that triangle is non-zero. I think this leads into an expansion of my second point, when doing lasso with a range of constraints/penalties there will be some that do not force any coefficients to 0. You can see this in a plot of the coefficients vs. the constraint/penalty. $\endgroup$
    – Greg Snow
    Commented Nov 23, 2015 at 19:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.