2
$\begingroup$

I have read that to apply ridge regression, we first need to standardize the predictive variables. That is because the variables should be in a homogenous scale so that lambda has an effect of the same magnitude on all of them.

However, taking a look at the ridge regression implementation of the scikit-learn library, I see that the X variables are centered but not standardized. That is, the variables retain their scale.

I was wondering if you know why that is, and when is each technique (standardization vs centering) advised.

Improve this question
$\endgroup$
4
  • $\begingroup$ The centering is probably to make life easier by orthogonalizing the variables against the intercept. For better or worse, it seems they're trusting the user to specify the right scales for ridge. In my view, you should basically always center and scale. $\endgroup$
    – John Madden
    Sep 21, 2022 at 13:24
  • $\begingroup$ Hi John, I missed the first part of your comment. Could you expand on what you mean? $\endgroup$
    – Paca
    Sep 21, 2022 at 14:00
  • $\begingroup$ Related: stats.stackexchange.com/questions/201909/… $\endgroup$
    – kjetil b halvorsen ♦
    Sep 21, 2022 at 14:11
  • $\begingroup$ @Paca there are little things that come up and bite your ankles if you're trying to write an algorithm and don't center your variables :) $\endgroup$
    – John Madden
    Sep 21, 2022 at 14:56

1 Answer 1

Reset to default
1
$\begingroup$

Looking at the source code is not necessarily the best way of learning such things. If you look at this discussion at Scikit-learn's GitHub (and outgoing links) its developers decided to deprecate a misleading default argument, so instead users could explicitly scale the data, as they suggest, with StandardScaler, i.e. subtracting the mean and dividing by standard deviation. So what you see in the code is a bad design decision that is currently being reverted. You are applying the same penalty to all the variables, so if they have different scales the penalty would impact them at different force.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Does that mean that the X variables should always be standardized? Is merely centering a bad idea? $\endgroup$
    – Paca
    Sep 21, 2022 at 13:58
  • $\begingroup$ @Paca yes, I edited for more explanation. $\endgroup$
    – Tim ♦
    Sep 21, 2022 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.