I have a categorical variable (e.g., office locations) with about 500 values. The frequency of the values follow a power-law distribution (if you sort the categorical values by frequency descendingly), where a few values have the highest frequencies, and the other ones fall off in frequency (long-tail, to the right).

I need to use this variable as a predictor in a logistic regression model. I am using scikit-learn. I want to know what's the best way to do one hot encoding (OHE) on this variable? I was thinking the following approaches.

  • Leave the top 10 frequently occurring values alone, but map all the other values as "other".
  • Since these values are geographic location, map the values into coarse-grained regions (e.g. go from cities to state/country/continent).

Are there any other sensible ways to transform this categorical variable for use in logistic regression? I also note that scikit-learn doesn't play well, or at all, with categorical variables, and that is why I have to use OHE as a preprocessing step. However, if there are other libraries or techniques out there that can handle categorical predictor variables, I'd be open to explore and try (even if they are available in R).

  • $\begingroup$ Think about using the factor variable with its 500 levels, but then use regularized logistic regression, aka tye lasso (in R, package glmnet). See stats.stackexchange.com/questions/146907/… for the idea of fused lasso (but I do not know about implementations in R) $\endgroup$ – kjetil b halvorsen Jun 7 '17 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.