26
$\begingroup$

For the LASSO (and other model selecting procedures) it is crucial to rescale the predictors. The general recommendation I follow is simply to use a 0 mean, 1 standard deviation normalization for continuous variables. But what is there to do with dummies?

E.g. some applied examples from the same (excellent) summer school I linked to rescales continuous variables to be between 0 and 1 (not great with outliers though), probably to be comparable to the dummies. But even that does not guarantee that the coefficients should be the same order of magnitude, and thus penalized similarly, the key reason for rescaling, no?

$\endgroup$
  • 2
    $\begingroup$ Short answer - no, do not rescale dummies $\endgroup$ – Affine Sep 9 '13 at 15:11
  • 4
    $\begingroup$ Related to this $\endgroup$ – julieth Sep 9 '13 at 15:14
  • $\begingroup$ @julieth, thanks a lot, let me know if you found some answers since. $\endgroup$ – László Sep 9 '13 at 15:21
26
$\begingroup$

According Tibshirani (THE LASSO METHOD FOR VARIABLE SELECTION IN THE COX MODEL, Statistics in Medicine, VOL. 16, 385-395 (1997)), who literally wrote the book on regularization methods, you should standardize the dummies. However, you then lose the straightforward interpretability of your coefficients. If you don't, your variables are not on an even playing field. You are essentially tipping the scales in favor of your continuous variables (most likely). So, if your primary goal is model selection then this is an egregious error. However, if you are more interested in interpretation then perhaps this isn't the best idea.

The recommendation is on page 394:

The lasso method requires initial standardization of the regressors, so that the penalization scheme is fair to all regressors. For categorical regressors, one codes the regressor with dummy variables and then standardizes the dummy variables. As pointed out by a referee, however, the relative scaling between continuous and categorical variables in this scheme can be somewhat arbitrary.

$\endgroup$
  • 3
    $\begingroup$ could you provide a precise reference to where Tibshirani suggests standardizing the dummies. $\endgroup$ – seanv507 Oct 18 '14 at 23:35
  • $\begingroup$ @seanv507 "...one codes the regressors with dummy variables and then standardizes the dummy variables". I think rocrat explanation is correct: in general you want all predictors, including dummies, to have the same scale and variance for the penalization to be fair. $\endgroup$ – Robert Kubrick May 30 '15 at 16:47
  • 1
    $\begingroup$ @RobertKubrick I disagree. The fundamental reason for regularisation is that small changes should have small effects. So the ideal case is that all your variables have a natural physical scale wrt your dependent variable and you do not normalise them. Typically we do not know the correct scale, so we resort to normalisation. However categorical variables have such a natural scale, namely the probability that they are 0 or 1: I would argue that a variable that is most of the time 0, is less important than a variable that flips between 0/1. Instead Jeff's answer seems appropriate. $\endgroup$ – seanv507 May 30 '15 at 18:57
8
$\begingroup$

Andrew Gelman's blog post, When to standardize regression inputs and when to leave them alone, is also worth a look. This part in particular is relevant:

For comparing coefficients for different predictors within a model, standardizing gets the nod. (Although I don’t standardize binary inputs. I code them as 0/1, and then I standardize all other numeric inputs by dividing by two standard deviation, thus putting them on approximately the same scale as 0/1 variables.)

$\endgroup$
  • 1
    $\begingroup$ And when he says "don’t standardize binary inputs", he seems to mean any one-hot group of variables, i.e. any dummies for categorical variables? $\endgroup$ – smci Feb 7 '17 at 16:10
  • 1
    $\begingroup$ True for the comparing coefficients (i.e. interpretation) bit, but from a regularization perspective that recommendation doesn't make as much sense. Reason is a binary variable has variance $p(1-p)$. For $p=0.5$ you get variance equal to $0.25$, which put it's on the same scale as the recommendation, but anything other than that you get progressively lower variance. It's just better to standardize for the optimization, and then report the coefficients in the original scale imo. $\endgroup$ – Firebug Jul 12 '18 at 12:31
  • $\begingroup$ what does he mean by "two standard deviation"? Is this x -> x / 2$\sigma$? $\endgroup$ – Alex Oct 15 '18 at 23:53
  • $\begingroup$ nevermind, it seems to be all explained here: stat.columbia.edu/~gelman/research/unpublished/… $\endgroup$ – Alex Oct 16 '18 at 0:01
3
$\begingroup$

This is more of a comment, but too long. One of the most used softwares for lasso (and friends) is R's glmnet. From the help page, printed by ?glmnet:

standardize: Logical flag for x variable standardization, prior to fitting the model sequence. The coefficients are always returned on the original scale. Default is ‘standardize=TRUE’. If variables are in the same units already, you might not wish to standardize. See details below for y standardization with ‘family="gaussian"’.

standardize is one of the arguments, defaults to true. So the $X$ variables are usually standardized, and this includes dummys (since there is no mention of an exception for them). But the coefficients are reported on the original scale.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.