# Categorical variable with a very large number of categories as a predictor

I am trying to use a categorical variable as a predictor in a supervised learning setting, but there are too many categories for the classification algorithm to handle, something like over a 1000 categories.

What are some ways to get a manageable number of categories, is there a standard way of binning these categories?

I suppose this binning should be performed on a training set disparate from a test set, to get a truer measure of out-of-sample error? If cross-validation is employed, I suppose the procedure should be run on on each fold.

## 2 Answers

Binning is really painful - many just say it's not a right thing to do, many others offer grouping by looking at the response, anyway you will feel a little uncomfortable:)

From the comments, I see that it's geographical data that you want to use as predictor (ZIP codes). Then consider kriging - I have used it for a similar problem - predicting the price of apartment from the address solely and was very satisfied, particularly since it solved one more important problem - predicting the outcome in case of a new predictor level (since the predictor becomes continuous rather than categorical, you will be able to predict the price of delivery even there was no delivery for a given ZIP). Beautiful heatmaps is another bonus.

Here is a nice lecture (with a nice Italian accent) by Fabio Veronesi with examples of kriging with R. http://www.fabioveronesi.net/r-course/lesson4.html

One problem that you will face is geocoding the GPS coordinates, and you can use the function geocode from ggmap package in R (using free service from Google up to some limit of queries per day, I queried for a week to get all done). http://cran.r-project.org/web/packages/ggmap/ggmap.pdf

Hopefully, this was useful, happy predicting.

Neural networks have been used with well over 1000 categories (see the Google paper on image recognition). But yes, it is not easy, and you may need much much more data to learn.

Have you considered aggregating categories into larger groups instead?

• This is really the heart of my question, what techinques can be used to accomplish aggregating categories into larger groups in a supervised environment? How will this technique affect the accuracy of the classifier?
– PT83
Feb 27, 2014 at 2:55
• Ideally, your application provides a hierarchy of classes, not a flat class scheme. Otherwise, what good is finding a super category, when your application needs a more detailed category instead? Feb 27, 2014 at 8:40
• And if there is no apriori hierarchy of classes? I dont know how detailed or not I need to get, this is what I need to determine. Let me give you an example, suppose I have zipcodes by state or region (like the Northeast), these are my categories. I want to predict some cost. What technique can I use to group these zipcodes so I can effectively use the groups as predictors for cost?
– PT83
Feb 27, 2014 at 14:48
• In this case, you are treating your zip codes an categorial input variable, not as output class variable. However, even with categorial variables one should usually try to first preprocess them into something more meaningful than a string of digits. Feb 27, 2014 at 16:10
• Right, into groups, and how should one do that to get a effective predictor for cost? Hierarchical clustering perhaps?
– PT83
Feb 27, 2014 at 17:52