10
$\begingroup$

Can someone please give me an example of when ridge would out perform lasso?

Won't lasso do better in most circumstances? If a regressor has a large coefficient, that means the regressor is a good predictor, so if we use ridge, we penalize that coefficient more. Isn't that bad?? Lasso will penalize the smaller coefficients more and the larger coefficients less because ridge squares the coefficient in the loss function (L2 norm), while lasso is just L1 norm.

We want less coefficients to prevent over-fitting, so wouldn't lasso ALWAYS be better?

Lasso will obviously be better if there are 2 good regressors and 15 bad ones, is there a canonical example of when ridge would outperform lasso?

Improve this question
$\endgroup$
1
  • 4
    $\begingroup$ Tibshirani's original paper where he introduces LASSO discusses a case (or some cases) when ridge beats LASSO. That could probably be considered canonical. $\endgroup$
    – Richard Hardy
    Jan 25, 2020 at 13:01

2 Answers 2

Reset to default
12
$\begingroup$

Both models penalise the inclusion of a non-zero coefficient, using a penalty function. LASSO regression penalises in a way that is proportionate to the absolute magnitude of the coefficient, and ridge regression penalises in a way that is proportionate to the square of the coefficient. Neither model penalises inputs in the likelihood function where the coefficient is set to zero. For inputs into the likelihood function where a coefficient is non-zero, LASSO regression penalises values near zero more heavily, whereas ridge regression penalises values far from zero more heavily. (In your question, you seem to be making the error of thinking that the squared value is always more than the absolute value. That is not true. For input values with magnitude less than one, the absolute value is larger than the square.)

Intuition tells us that ridge regression will tend to outperform LASSO regression in cases where the true non-zero coefficients are close to zero, relative to the noise in the regression. In this case, ridge regression penalises these values less, so it is more likely to estimate non-zero values for these coefficients. LASSO regression penalises these coefficients more, so it is more likely to incorrectly estimate them to be zero. On the basis of this intuition, I would recommend that you compare these models for some data generated from a regression with coefficients that are small relative to the noise in the regression. If you were to conduct a simulation study with cases like this, you should find that ridge regression tends to outperform LASSO in these cases.

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ The canonical 1996 LASSO paper noted by @RichardHardy in a comment on this question is linked from this page. Consistent with this answer, the example in which ridge outperformed LASSO was the case with the lowest signal-to-noise ratio and with no true coefficient values of 0. $\endgroup$
    – EdM
    Jan 25, 2020 at 22:37
  • $\begingroup$ @EdM Very cool link. $\endgroup$
    – Frank
    Jan 27, 2020 at 1:32
1
$\begingroup$

Ridge was originally designed for correlated variables, and that's where it's best.

Consider examinations to determine a degree. ( Which supposedly is measuring ability)

Which do you think is more reliable: taking the average of all the exams or picking a single exam most correlated with ability (if there is one)? Averaging over the different exams removes (independent noise - you didn't sleep well one day etc)

Ridge will take the average of these correlated inputs (IE the separate exams), whilst lasso will just select one.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Lasso is just a one norm instead of a two norm. In your example, if a regressor is "Hours spent studying," then maybe it has a huge coefficient because it is the most important regressor, and maybe lose sleep is really insignificant in comparison. Wouldn't ridge, being a two norm, be harsher on the "study time" regressor, since it is two norm (quadratic penalty), meaning it is harder on regressors with larger coefficients? Isn't ridge bad for what we want, which is highly correlated regressors, and better when you have a bunch of not so good regressors? $\endgroup$
    – Frank
    Jan 29, 2020 at 4:27
  • $\begingroup$ I meant that the regressors are correlated amongst themselves (as well as dependent variable). So a bunch of maths exams - rather than putting a large coefficient on a single maths exam it will put a small coefficient on each of them [which add up to the same overall effect]. $\endgroup$
    – seanv507
    Jan 29, 2020 at 18:27
  • $\begingroup$ I agree that if you have a couple of major variables then lasso is better. But that is not always the case. I was explaining other situations. $\endgroup$
    – seanv507
    Jan 29, 2020 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.