28
$\begingroup$

Hey guys I found one or two papers which use ridge regression (for basketball data). I was always told to standardize my variables if I ran a ridge regression, but I was simply told to do this because ridge was scale variant (ridge regression wasn't really part of our course, so our lecturer skimmed through it).

These papers I read didn't standardize their variables, which I found a bit surprising. They also ended up with large values of lambda (around the 2000-4000 level) via cross-validation, and I was told that this is due to not standardizing the variables.

How exactly does leaving the variable(s) unstandardised lead to high lambda values and also, what are the consequences of not standardizing the variables in general? Is it really such a big deal?

Any help is much appreciated.

$\endgroup$

2 Answers 2

31
$\begingroup$

Ridge regression regularize the linear regression by imposing a penalty on the size of coefficients. Thus the coefficients are shrunk toward zero and toward each other. But when this happens and if the independent variables does not have the same scale, the shrinking is not fair. Two independent variables with different scales will have different contributions to the penalized terms, because the penalized term is a sum of squares of all the coefficients. To avoid such kind of problems, very often, the independent variables are centered and scaled in order to have variance 1.

[Later edit to answer to comment]

Suppose now that you have an independent variable $height$. Now, human height might be measured in inches or meters or kilometers. If measured in kilometers, than in standard linear regression I think it will give a much bigger coefficient term, than if measured in millimeters.

The penalization term with lambda is the same as expressing the square loss function with respect to the sum of squared coefficients less than or equal a given constant. That means, bigger lambda gives much space to the squared sum of coefficients, and lower lambda a smaller space. Bigger or smaller space means bigger or smaller absolute values of the coefficients.

By not using standardization, then to fit the model might require big absolute values of the coefficients. Of course, we might have a big coefficient value naturally, due to the role of the variable in the model. What I state is that this value might have an artificially inflated value due to not scaling. So, scaling also decreases the need for a big values of coefficients. Thus, the optimal value of lambda would be usually smaller, which corresponds to a smaller sum of squared values of coefficients.

$\endgroup$
4
  • $\begingroup$ Thanks. How would not standardizing lead to a higher estimated test-error (via cross validation), and thus a higher lambda needed, though? $\endgroup$
    – l_davies93
    Commented Aug 7, 2014 at 11:30
  • $\begingroup$ I appended my thought in the answer $\endgroup$
    – rapaio
    Commented Aug 7, 2014 at 12:25
  • $\begingroup$ I know that this is an old Question, but could you explain maybe why the Tuning Parameter should getting bigger, if we transform our data from kilometers to meters for example $\endgroup$
    – Leo96
    Commented Aug 13, 2018 at 14:46
  • $\begingroup$ I guess as you go from Km to meters the values will become big as its x1000, hence coefficient will become smaller in the model and which in turn will make the penalty term very small, so to have an optimal significance of penalty term, the tunning parameter will have to be increased. $\endgroup$
    – jazz
    Commented May 4, 2022 at 13:05
8
$\begingroup$

Though four years late, hope someone will benefit from this.... The way I understood it, coeff is how much target variable changes for a unit change in independent variable (dy / dx). Let us assume we are studying relation between weight and height and weight is measured in Kg. When we use Kilometers for height, you can imagine most of the data points (for human height) packed closely. Thus, for a small fractional change in height there will be huge change in weight (assuming weight increase with height). The ratio dy /dx will be huge. On the other hand, if height is measured in millimeters, data will be spread far and wide on the height attributes. A unit change in height will have no significant change in weight dy /dx will be very small almost close to 0. Lambda will have to be higher when height is in Km compared to lambda when height is in millimeters

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.