It is generally agreed that in penalized regression models, such as ridge regression, the lasso, and the elastic net, one should standardize the predictors (such as dividing each by its SD) so that the penalty applies equally to the different predictors, instead of applying to different degrees based on the scale of the predictors. But should one do this even for binary predictors, which are coded as 0 and 1?

To be more specific, suppose my goal is maximizing the predictive accuracy (i.e., minimizing the test error) of my model, and I have little a priori knowledge about the relative importance of the predictors. Which is likely to get better predictive accuracy, rescaling my binary predictors along with my continuous predictors, or leaving them alone?

A frequently mentioned paper in the context of predictor standardization and binary predictors is the following, but it doesn't seem to consider the case of penalized regression.

Gelman, A. (2008). Scaling regression inputs by dividing by two standard deviations. Statistics in Medicine, 27, 2865–2873. doi:10.1002/sim.3107. http://www.stat.columbia.edu/~gelman/research/published/standardizing7.pdf

A similar but more general question is: Are categorical variables standardized differently in penalized regression?

Edit: a more closely related question is: whether to rescale indicator / binary / dummy predictors for LASSO

  • 1
    $\begingroup$ One way I think about it is in terms of prior belief. If I have a bunch of binary predictors, and my prior belief is that each of them has an equal chance of being included or excluded, then the regularization penalty should apply equally to all of them. This requires standardization. I'm not sure there is a way to make this rigorous, but it's a useful way to think about the problem. $\endgroup$ – Matthew Drury May 4 '15 at 18:46
  • $\begingroup$ Another way is in terms of the units of the regularization adjustment to the loss. If you don't standardize, then there is no consistent way to assign units. I'm not sure this helps in the dummy indicator case though. $\endgroup$ – Matthew Drury May 4 '15 at 18:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.