10
$\begingroup$

In my attempt to reason about it intuitively I am concluding that ridge might be more robust to outliers.

Following is my intuitive/lose reasoning :

If there is an outlier then to match my prediction to it I might increase weight value on some dimension, and when I do that ridge will penalize it more compared to Lasso and not let it take a higher value. So it seems like ridge is more robust, but most people say that Lasso is more robust to outliers.

So my question is that what is wrong in my thought process, and what is the correct intuitive way to think about it ?

Improve this question
$\endgroup$
2
  • $\begingroup$ I have exactly the same thoughts as yours. Have you figured it out ? $\endgroup$
    – user152503
    Dec 6, 2020 at 10:18
  • $\begingroup$ no not yet, didn't have the time to think further on it and didn't find any resource that addresses this issue properly $\endgroup$
    – Siddharth Shakya
    Dec 7, 2020 at 12:14

1 Answer 1

Reset to default
5
$\begingroup$

Let's first consider what an outlier does to the coefficients:

  • If it has low leverage, nothing;
  • If it has high leverage, it pulls the coefficient towards itself (either increasing or decreasing it).

When you apply the LASSO penalty to OLS, you penalize the coefficients by summing their absolute values. An outlier with sufficient leverage increases/decreases a coefficient, also affecting the penalty linearly. This will somewhat increase/decrease the penalty to the the other coefficients, but not by much.

When you apply the ridge penalty, the sum of squared coefficients shrinks the coefficient. This means that outlyingness will not only increase the OLS quadratically, but also the penalty. As such, all the other coefficients might be shrunk considerably more/less (depending on what kind of outlier you're dealing with).

This sensitivity of the penalty to changes in the coefficients (and thus to outliers) means that ridge is less robust to outliers than LASSO.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Isn't shrinking the coefficient good for robustness against outliers, as we aren't allowed to increase/decrease the weight by large amount. You have only explained what happens and how they behave , my thinking is that, this behavior would make ridge robust. I am essentially asking that what is wrong in this thinking ? $\endgroup$
    – Siddharth Shakya
    Apr 23, 2019 at 12:15
  • 2
    $\begingroup$ Greater sensitivity to outliers is identical in meaning to lower robustness to outliers. Whether this sensitivity is good, or desired, depends on your intended use of the model. $\endgroup$
    – Frans Rodenburg
    Apr 23, 2019 at 13:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.