11
$\begingroup$

I am aware it is common practice to standardize the features for ridge and lasso regression, however, would it ever be more practical to normalize the features on a (0,1) scale as an alternative to z-score standardization for these regression methods?

$\endgroup$
4
$\begingroup$

If you apply normalisation (squeeze in [0,1]) you will have a measure of relative variable importance but it will change the scale of your variables and you will lose all model interpretability. The advantage of standardisation is that you can still interpret the model as you would with unregularised OLS regression (this has already been already answered here).

| cite | improve this answer | |
$\endgroup$
  • 3
    $\begingroup$ Regularised model is acting very differently with or without normalization !! Specifically, if we do not normlize features, we will have different penalties on different features! $\endgroup$ – Haitao Du Jun 26 '17 at 14:11
  • 1
    $\begingroup$ I was specifically talking about Lasso coefficient interpretation, not estimation. Given that the estimates would change, I would be curious to know how would model interpretation change. $\endgroup$ – Digio Jun 27 '17 at 14:34
  • 1
    $\begingroup$ It doesn't seem to me that the question you link to in your answer supports the point you're making. Could you make more explicit in your original post why the interpretation of ols coefficients agrees with lasso coefficients only when the features are standardized? Thank you! $\endgroup$ – user795305 Oct 22 '17 at 13:38
  • $\begingroup$ @Ben, you misunderstood my answer (my fault perhaps). The answer I've linked to explains how model coefficients in lasso and in simple regression (OLS or otherwise) are interpreted in the same way - under any circumstances (standardised or not). With normalisation (in any type or parametric regression), you lose the original scale and you cannot interpret the coefficients without back-transforming. With standardisation, you interpret the model in the normal manner. $\endgroup$ – Digio Oct 23 '17 at 9:47
  • $\begingroup$ @Digio: why would normalization make the coefficients any less interpretable than any other scale change? Aren't normalization and standardization identical transformations but for the numerator they use? Both require back transformation in the same way to interpret the coefficients as dV/dx, with x in the units of the un-scaled features. Yes, coeffs of standardized features can also be directly interpreted as dV/dz, z being the z-score of each feature, and this interpretation isn't possible with normalization. But does that mean that coeffs of normalized features lose all interpretation? $\endgroup$ – OldSchool May 20 at 14:33
0
$\begingroup$

Normalization is very important for methods with regularization. This is because the scale of the variables affect the how much regularization will be applies to specific variable.

For example, suppose one variable is in a very large scale, say order of millions and another variable is from 0 to 1. Then, we can think the regularization will have little effect on first variable.

As well as we do normalization, normalize it to 0 to 1 or standardize the features does not matter too much.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ This answer is stating the obvious. By "normalisation" here it is meant squeezing all values in [0,1], it's not just another word for standardisation. The question is about the effects of normalization in [0,1] vs. standardization ~N(0,1) on model coefficients. $\endgroup$ – Digio Jun 27 '17 at 20:07
  • $\begingroup$ What does it mean to normalize to [0,1]? There are many ways to achieve that. What exactly is your recommendation for penalized regression? $\endgroup$ – Cagdas Ozgenc Jun 28 '17 at 8:48
  • 1
    $\begingroup$ As the question states to "normalize the features on a (0,1)" scale, though maybe feature rescaling is a better term, is a general technique to produce coefficient estimates that express relative variable importance (similar to RF's purity measure). Yes, there are many ways to achieve this and it is not something specific to penalised regression but this question is about the effect of feature rescaling (not standardisation) on Lasso. $\endgroup$ – Digio Jun 29 '17 at 7:21
  • $\begingroup$ what do you mean by "normalize it to 0 to 1 or standardize the features does not matter too much"? In what sense does it not matter too much? Could you provide any intuition or references for this claim? $\endgroup$ – user795305 Oct 12 '17 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.