In penalized/regularized regression (lasso, ridge, etc.) the predictors are typically standardized to be centered at 0 and often to have variance 1. Are categorical predictors treated differently. If so, why? What are the consequences of using the same standardization? Is a reference available?

  • 1
    $\begingroup$ Andrew Gelman has a set of unpublished recommendations about standardizing categorical variables here. I may write up a longer answer if I have time. $\endgroup$ – David J. Harris Sep 9 '13 at 21:14
  • $\begingroup$ Thanks; that looks interesting. But categorical variables aren't only binary; I think the answer will need to cover the grouped lasso. $\endgroup$ – Scortchi - Reinstate Monica Sep 9 '13 at 21:19
  • $\begingroup$ The paper deals with other kinds of categories as well (e.g. ethnicity, neighborhood). The discussion about multilevel models should also have direct applications for the group lasso. $\endgroup$ – David J. Harris Sep 9 '13 at 21:27
  • 1
    $\begingroup$ I'm not an expert at all, but Hastie et al. in The Elements of Statistical Learning (2nd ed), Chapter 3.8.4 when discussing grouped lasso, indicates that it may be an idea to group categorical predictors together. $\endgroup$ – Avraham Sep 9 '13 at 21:49
  • $\begingroup$ I believe the motivation for standardization for penalized regression (ensuring covariates are affected similarly) differs from that of the setting Gelman discusses. I have been researching this question and got some good tips from Frank Harrell's book. I will respond after doing some more work if nobody else has responded. $\endgroup$ – julieth Sep 9 '13 at 23:05

I think the main point is what you want to do with the model. There is not a single answer to whether you should standardize none, some or all of variables. It depends on what you want your model for.

Using the z-score of the predictors (what you call standardizing), puts all the predictors in the same scale, but makes interpretation a little bit more difficult. The interpretation of the coefficients is now "how much a change in the standard deviation affects the output variable".

Many times, penalized/regularized regressions are not suitable for interpretation, because you are introducing a bias in the coefficients. Usually when you use such models, you are interested in the predictions, not in doing a counterfactual analysis. Standardizations are useful because they make the problem numerically more stable. If such is your case, it doesn't make a big difference if you "standardize" your categorical predictors or not.

Try asking a more specific answer, including what kind of analysis you want to do with your problem, and you can get a more specific answer :)


Not the answer you're looking for? Browse other questions tagged or ask your own question.