25
$\begingroup$

I have read three main reasons for standardising variables before something such as Lasso regression:

1) Interpretability of coefficients.

2) Ability to rank the coefficient importance by the relative magnitude of post-shrinkage coefficient estimates.

3) No need for intercept.

But I am wondering about the most important point. Do we have reason to think that standardisation would improve the out of sample generalisation of the model? Also I don't care if I don't need an intercept in my model; adding one doesn't hurt me.

$\endgroup$
  • 1
    $\begingroup$ Clarification: you seem to want to ask, "Provided that standardization is optional (one of the special cases where the results are not skewed by different magnitudes), then will standardization improve out-of-sample generalization?" Is this correct? $\endgroup$ – Drew75 Feb 13 '14 at 15:14
  • $\begingroup$ @Drew75 I prefer a breakdown of cases e.g. does it help when the results are "skewed by different magnitudes", does it help when the results aren't skewed, et cetera, the best answer will cover different situations. $\endgroup$ – Jase Feb 13 '14 at 16:33
  • 1
    $\begingroup$ Then your question isn't about Lasso (because in general standardization is necessary before Lasso). It's more general. Perhaps change the title and the first sentence of the question. $\endgroup$ – Drew75 Feb 13 '14 at 18:25
  • $\begingroup$ @Drew: That's rather question-begging: Why's it necessary (when isn't it?)? What does it mean to skew the results (compared to what?)? I think the question's fine as it stands. $\endgroup$ – Scortchi Feb 13 '14 at 20:46
  • $\begingroup$ @Drew75 My question is about Lasso. $\endgroup$ – Jase Feb 14 '14 at 3:31
18
$\begingroup$

Lasso regression puts constraints on the size of the coefficients associated to each variable. However, this value will depend on the magnitude of each variable. It is therefore necessary to center and reduce, or standardize, the variables.

The result of centering the variables means that there is no longer an intercept. This applies equally to ridge regression, by the way.

Another good explanation is this post: Need for centering and standardizing data in regression

$\endgroup$
  • $\begingroup$ This is either not an answer or an extremely indirect answer to my question. Please explain the link between your answer and out of sample generalisation (which was the question). $\endgroup$ – Jase Feb 13 '14 at 8:57
  • 10
    $\begingroup$ @Jase: It does address the main reason for standardization, which you omitted from your list: if you want to drop predictors with small coefficients (or otherwise use a penalty term depending on coefficient magnitude), you need to decide what counts as "small". Though standardization isn't mandatory before LASSO or other penalized regression methods, it's rarely the case that the original scales the predictors happen to be measured in are useful for this purpose. $\endgroup$ – Scortchi Feb 13 '14 at 9:36
  • 3
    $\begingroup$ And the point about centring is that you don't usually want to drop or shrink the intercept. $\endgroup$ – Scortchi Feb 13 '14 at 9:38
  • 2
    $\begingroup$ @Jase: Yes that's what I mean (assuming $\lambda$'s the shrinkage parameter). And whether a coefficient estimate's among the tiniest (however you choose \lambda$) depends on whether it's measured in kilometres, micrometres, the no. standard deviations from its mean value in the sample, or some other unit. From a Bayesian viewpoint you're putting weakly informative priors over the true coefficient values, not uninformative ones. $\endgroup$ – Scortchi Feb 13 '14 at 14:36
  • 2
    $\begingroup$ Very broadly, how much you shrink overall is going to affect generalization to random hold-out samples; the somewhat arbitrary decision how much to shrink each predictor relative to the others is going to affect generalization to new samples from similar populations, where the coefficients are a bit different, where the distribution of predictors isn't necessarily much like that in the training set, &c. (Of course your question deserves a more fully thought-out answer.) $\endgroup$ – Scortchi Feb 13 '14 at 14:37
1
$\begingroup$

The L1 penalty parameter is a summation of absolute beta terms. If the variables are all of different dimensionality then this term is really not additive even though mathematically there isn't any error.

However, I don't see the dummy/ categorical variables suffering from this issue and think they need not be standardized. standardizing these may just reduce interpretability of variables

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.