4
$\begingroup$

I wonder whether the response variable needs to be normalized before LASSO estimation (I am using the lars package in R to perform LASSO estimation). My guess is that only right-hand side variables need to be normalized, is this correct?

For the sake of interpretation, I would prefer to only normalize right-hand side variables and leave out the response variable. If this is fine, I guess I need to normalize the data myself as the argument 'normalize' in the lars package performs normalization on all variables including the response variable - at least I believe so.

Thanks for help!

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
4
$\begingroup$

You don't need to normalize the response variable, as that just scales the error term that the procedure minimizes.

It doesn't seem that lars normalizes the response variable, anyway. Examine the code by loading the package and typing "lars" (without quotes or parentheses) at the R prompt. As I read the code, if "normalize=TRUE" then only the x (predictor/"right-hand") variables are normalized. The y (response) variable is untouched. I suspect that the statement in the documentation that "each variable is standardized to have unit L2 norm" was only meant to apply to the x values.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ As an FYI, glment does standardize $y$ when doing linear regression (elastic net). $\endgroup$
    – Matthew Drury
    Commented Jun 8, 2015 at 17:14
  • $\begingroup$ Many thanks for your swift response!!! After looking at the source code, I agree with you that "normalize=TRUE" applies to predictors only. $\endgroup$
    – kanimbla
    Commented Jun 8, 2015 at 17:14
1
$\begingroup$

You don't 'need' to do anything, but it is highly recommended to normalize the variables. This is because lasso is a technique that penalizes large coefficients. If the variables are on different scales, the technique will tend to just penalize variables on the smallest scale as they will have the largest coefficient. For example if y~x and z, and w are noise then lasso will just give the (correct) model y~x. However if x1 = 10^{-100}x then the right model for predicting y in terms of x1,z,w would be y~(10^{100})x1. However if the coefficients are constrained to be less than that in absolute value, lasso can't give the correct model. The more that the coefficients are constrained, the worse the model will be. You can imagine that if one is predicting y on two (equally important) variables, the same sort of issue will occur.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for further clarification regarding normalization within the LASSO context, very useful comments! $\endgroup$
    – kanimbla
    Commented Jun 8, 2015 at 17:18
  • $\begingroup$ then question is about normalizing the response variable (y) not the features (x) $\endgroup$
    – Areza
    Commented Jun 14, 2021 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.