I have a categorical variable (e.g., office locations) with about 500 values. The frequency of the values follow a power-law distribution (if you sort the categorical values by frequency descendingly), where a few values have the highest frequencies, and the other ones fall off in frequency (long-tail, to the right).

I need to use this variable as a predictor in a logistic regression model. I am using scikit-learn. I want to know what's the best way to do one hot encoding (OHE) on this variable? I was thinking the following approaches.

  • Leave the top 10 frequently occurring values alone, but map all the other values as "other".
  • Since these values are geographic location, map the values into coarse-grained regions (e.g. go from cities to state/country/continent).

Are there any other sensible ways to transform this categorical variable for use in logistic regression? I also note that scikit-learn doesn't play well, or at all, with categorical variables, and that is why I have to use OHE as a preprocessing step. However, if there are other libraries or techniques out there that can handle categorical predictor variables, I'd be open to explore and try (even if they are available in R).

  • $\begingroup$ Think about using the factor variable with its 500 levels, but then use regularized logistic regression, aka tye lasso (in R, package glmnet). See stats.stackexchange.com/questions/146907/… for the idea of fused lasso (but I do not know about implementations in R) $\endgroup$ – kjetil b halvorsen Jun 7 '17 at 14:53
  • $\begingroup$ Would your intended use of your model allow you to treat the office locations as a random effect instead of a multi-category fixed effect? That is sometimes done to account for differences among medical institutions in analysis of multi-center data. $\endgroup$ – EdM Jan 25 at 20:21

You could use the categorical variable with its 500 levels as is, but then use regularized logistic regression. In Principled way of collapsing categorical variables with many levels? one idea is to used the fused lasso, but there are other possibilities. I cannot see how the power-law distribution is relevant, and your idea of merging all but the 10 most frequent levels is problematic. That two levels both have low frequency in no way indicates they have the same effect!

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.