My question relates on the Ridge vs Lasso Regression. I know the difference in the cost function (ridge penalizes sum of quadratic coefficients, lasso penalizes sum of absolute value of coefficients). Moreover, I also know that Lasso is able of reducing some coefficients completely to zero while ridge only does towards zero.

So my question is whether one can therefore say from a theoretical perspective that Lasso should have a lower variance (generalizes better) but a higher bias than Ridge because of the above mentioned property of reducing coefficients completely to zero (of course if one applies the same strength of regularization for both of them)?

Thank you.


1 Answer 1



There are several issues with the way you describe the premise of the question. For one, it is meaningless to say "the same strength of regularization." The fact that you may use the same greek letter for regularization parameters in ridge and lasso doesn't make them directly comparable. Just use $\alpha$ in ridge and $\lambda$ in lass for regularization paramater to see my point. You can drive coefficients to zero with both methods by cranking up the regularization.

Secondly, you seem to be trying to look at these techniques from variance/bias point of view, which is not natural in this context. Regularization in these techniques is to address overparameterization. They do it slightly differently, where ridge is rapidly penalizing the coefficients as they grow, and lasso does it uniformly over the value ranges of coefficients. You may say lasso may allow more coefficients to stick out farther compared to ridge.

  • $\begingroup$ Okay got that. But I do not understand why one can not make the two comparable by setting the same regularization strength? Both parameters mean "economically" regularization strength and describe therefore the same. Moreover, I understand that you can bring the coefficients to zero with Ridge. But my argument was that is goes only towards zero and not makes them exactly zero, therefore why can one not say that LASSO makes simpler models because of this property? Thanks for clarification. $\endgroup$
    – alphaH
    Commented Oct 11, 2021 at 19:26
  • $\begingroup$ @alphaH, the parameters belong to different functions with different units of measure even. Suppose, you regressed weight on height of a student, then in $\alpha\times h_i^2$ term $\alpha\sim 1/cm^2$ while in $\lambda|h_t|$ term $\lambda\sim 1/cm$. It's like comparing the "strength" of coefficient of linear and quadratic terms in a polynomial, it makes no sense $\endgroup$
    – Aksakal
    Commented Oct 11, 2021 at 19:50
  • $\begingroup$ Okay thank you. So then as far as I understand you one can not really draw a senseful conclusion/comparison between Lasso vs Ridge when it comes to bias vs variance. Its really data dependent. $\endgroup$
    – alphaH
    Commented Oct 11, 2021 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.