4
$\begingroup$

I wonder whether the response variable needs to be normalized before LASSO estimation (I am using the lars package in R to perform LASSO estimation). My guess is that only right-hand side variables need to be normalized, is this correct?

For the sake of interpretation, I would prefer to only normalize right-hand side variables and leave out the response variable. If this is fine, I guess I need to normalize the data myself as the argument 'normalize' in the lars package performs normalization on all variables including the response variable - at least I believe so.

Thanks for help!

$\endgroup$
0
3
$\begingroup$

You don't need to normalize the response variable, as that just scales the error term that the procedure minimizes.

It doesn't seem that lars normalizes the response variable, anyway. Examine the code by loading the package and typing "lars" (without quotes or parentheses) at the R prompt. As I read the code, if "normalize=TRUE" then only the x (predictor/"right-hand") variables are normalized. The y (response) variable is untouched. I suspect that the statement in the documentation that "each variable is standardized to have unit L2 norm" was only meant to apply to the x values.

$\endgroup$
2
  • 2
    $\begingroup$ As an FYI, glment does standardize $y$ when doing linear regression (elastic net). $\endgroup$ – Matthew Drury Jun 8 '15 at 17:14
  • $\begingroup$ Many thanks for your swift response!!! After looking at the source code, I agree with you that "normalize=TRUE" applies to predictors only. $\endgroup$ – kanimbla Jun 8 '15 at 17:14
0
$\begingroup$

You don't 'need' to do anything, but it is highly recommended to normalize the variables. This is because lasso is a technique that penalizes large coefficients. If the variables are on different scales, the technique will tend to just penalize variables on the smallest scale as they will have the largest coefficient. For example if y~x and z, and w are noise then lasso will just give the (correct) model y~x. However if x1 = 10^{-100}x then the right model for predicting y in terms of x1,z,w would be y~(10^{100})x1. However if the coefficients are constrained to be less than that in absolute value, lasso can't give the correct model. The more that the coefficients are constrained, the worse the model will be. You can imagine that if one is predicting y on two (equally important) variables, the same sort of issue will occur.

$\endgroup$
2
  • $\begingroup$ Thanks for further clarification regarding normalization within the LASSO context, very useful comments! $\endgroup$ – kanimbla Jun 8 '15 at 17:18
  • $\begingroup$ then question is about normalizing the response variable (y) not the features (x) $\endgroup$ – user4581 Jun 14 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.