One of the predictors I had in a logit model is "City". Problem is this categorical variable has too many factor levels. e.g. In a Sample of $\sim 3000$ there are already $\sim 200$ different cities.

Is it fair to still retain City as a predictor or should I purge it entirely from the model? An alternative is to retain, say, the top five most common cities and then code all the rest as a new factor level "Others". The top city occurs $\sim 60$ times but by the fifth common city this occurance drops down to 30. Some cities occur only once or twice in the dataset.

PS. One problem I face (if I retain all factor levels) is that certain levels occur in the validation set but not in the training set. Then the model complains at validation time.

PPS. On more reading, I found sugesstions to use combine.levels() from the Hmisc package in R. Maybe that will work, though not sure how exactly yet.

Is there an elegant way to deal with these issues?

  • 1
    $\begingroup$ Have you considered replacing the ordinal City variable with characteristics of those cities? In particular, ones that are theoretically related to your model? For example, you might replace City with population density, if that's related to the outcome you're trying to predict. This would require finding the relevant data and joining to your current data; but, if your validation set has a new city, it won't throw an error so long as you can find the relevant characteristics. $\endgroup$ Commented Apr 21, 2014 at 14:15
  • $\begingroup$ Have a look at this: stats.stackexchange.com/questions/25324/… and this win-vector.com/blog/2012/07/… $\endgroup$
    – B_Miner
    Commented Apr 21, 2014 at 17:19
  • 1
    $\begingroup$ The applicability of this depends on how of course you are using the predictor. If this is your main variable for inference, it probably doesnt work and I would collapse into an other group as you said. If the main goal is for prediction or if this is a control variable, it can work. $\endgroup$
    – B_Miner
    Commented Apr 21, 2014 at 17:21