47
$\begingroup$

For the LASSO (and other model selecting procedures) it is crucial to rescale the predictors. The general recommendation I follow is simply to use a 0 mean, 1 standard deviation normalization for continuous variables. But what is there to do with dummies?

E.g. some applied examples from the same (excellent) summer school I linked to rescales continuous variables to be between 0 and 1 (not great with outliers though), probably to be comparable to the dummies. But even that does not guarantee that the coefficients should be the same order of magnitude, and thus penalized similarly, the key reason for rescaling, no?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ Short answer - no, do not rescale dummies $\endgroup$
    – Affine
    Commented Sep 9, 2013 at 15:11
  • 4
    $\begingroup$ Related to this $\endgroup$
    – julieth
    Commented Sep 9, 2013 at 15:14
  • $\begingroup$ @julieth, thanks a lot, let me know if you found some answers since. $\endgroup$
    – László
    Commented Sep 9, 2013 at 15:21

3 Answers 3

Reset to default
40
$\begingroup$

According Tibshirani (THE LASSO METHOD FOR VARIABLE SELECTION IN THE COX MODEL, Statistics in Medicine, VOL. 16, 385-395 (1997)), who literally wrote the book on regularization methods, you should standardize the dummies. However, you then lose the straightforward interpretability of your coefficients. If you don't, your variables are not on an even playing field. You are essentially tipping the scales in favor of your continuous variables (most likely). So, if your primary goal is model selection then this is an egregious error. However, if you are more interested in interpretation then perhaps this isn't the best idea.

The recommendation is on page 394:

The lasso method requires initial standardization of the regressors, so that the penalization scheme is fair to all regressors. For categorical regressors, one codes the regressor with dummy variables and then standardizes the dummy variables. As pointed out by a referee, however, the relative scaling between continuous and categorical variables in this scheme can be somewhat arbitrary.

Improve this answer
$\endgroup$
6
  • 3
    $\begingroup$ could you provide a precise reference to where Tibshirani suggests standardizing the dummies. $\endgroup$
    – seanv507
    Commented Oct 18, 2014 at 23:35
  • 1
    $\begingroup$ @seanv507 "...one codes the regressors with dummy variables and then standardizes the dummy variables". I think rocrat explanation is correct: in general you want all predictors, including dummies, to have the same scale and variance for the penalization to be fair. $\endgroup$
    – Robert Kubrick
    Commented May 30, 2015 at 16:47
  • 1
    $\begingroup$ @RobertKubrick I disagree. The fundamental reason for regularisation is that small changes should have small effects. So the ideal case is that all your variables have a natural physical scale wrt your dependent variable and you do not normalise them. Typically we do not know the correct scale, so we resort to normalisation. However categorical variables have such a natural scale, namely the probability that they are 0 or 1: I would argue that a variable that is most of the time 0, is less important than a variable that flips between 0/1. Instead Jeff's answer seems appropriate. $\endgroup$
    – seanv507
    Commented May 30, 2015 at 18:57
  • $\begingroup$ Related discussion in a scope of feature importance and grouped LASSO here: stats.stackexchange.com/a/360473/60250 $\endgroup$
    – m-dz
    Commented Feb 19, 2020 at 11:23
  • $\begingroup$ "You are essentially tipping the scales in favor of your continuous variables (most likely)" - I assume that this assumes that continuous variables are likely larger than binaries? If so, don't larger variables end up with smaller coefficients, and are thus more likely to be removed under Lasso? $\endgroup$
    – James
    Commented Oct 14, 2020 at 15:54
13
$\begingroup$

Andrew Gelman's blog post, When to standardize regression inputs and when to leave them alone, is also worth a look. This part in particular is relevant:

For comparing coefficients for different predictors within a model, standardizing gets the nod. (Although I don’t standardize binary inputs. I code them as 0/1, and then I standardize all other numeric inputs by dividing by two standard deviation, thus putting them on approximately the same scale as 0/1 variables.)

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ And when he says "don’t standardize binary inputs", he seems to mean any one-hot group of variables, i.e. any dummies for categorical variables? $\endgroup$
    – smci
    Commented Feb 7, 2017 at 16:10
  • 6
    $\begingroup$ True for the comparing coefficients (i.e. interpretation) bit, but from a regularization perspective that recommendation doesn't make as much sense. Reason is a binary variable has variance $p(1-p)$. For $p=0.5$ you get variance equal to $0.25$, which put it's on the same scale as the recommendation, but anything other than that you get progressively lower variance. It's just better to standardize for the optimization, and then report the coefficients in the original scale imo. $\endgroup$
    – Firebug
    Commented Jul 12, 2018 at 12:31
  • $\begingroup$ what does he mean by "two standard deviation"? Is this x -> x / 2$\sigma$? $\endgroup$
    – Alex
    Commented Oct 15, 2018 at 23:53
  • $\begingroup$ nevermind, it seems to be all explained here: stat.columbia.edu/~gelman/research/unpublished/… $\endgroup$
    – Alex
    Commented Oct 16, 2018 at 0:01
  • $\begingroup$ @Alex It seems like this is just helpful for explanation. It is still to make std equal among variables. But instead of standardize all to 1 std, you now only scale numeric variables to match the std of dummy variables. $\endgroup$
    – lovetl2002
    Commented Apr 28, 2023 at 3:26
6
$\begingroup$

This is more of a comment, but too long. One of the most used softwares for lasso (and friends) is R's glmnet. From the help page, printed by ?glmnet:

standardize: Logical flag for x variable standardization, prior to fitting the model sequence. The coefficients are always returned on the original scale. Default is ‘standardize=TRUE’. If variables are in the same units already, you might not wish to standardize. See details below for y standardization with ‘family="gaussian"’.

standardize is one of the arguments, defaults to true. So the $X$ variables are usually standardized, and this includes dummys (since there is no mention of an exception for them). But the coefficients are reported on the original scale.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.