Categorical variable with a very large number of categories as a predictor I am trying to use a categorical variable as a predictor in a supervised learning setting, but there are too many categories for the classification algorithm to handle, something like over a 1000 categories.  
What are some ways to get a manageable number of categories, is there a standard way of binning  these categories?  
I suppose this binning should be performed on a training set disparate from a test set, to get a truer measure of out-of-sample error? If cross-validation is employed, I suppose the procedure should be run on on each fold.
 A: Binning is really painful - many just say it's not a right thing to do, many others offer grouping by looking at the response, anyway you will feel a little uncomfortable:)
From the comments, I see that it's geographical data that you want to use as predictor (ZIP codes). Then consider kriging - I have used it for a similar problem - predicting the price of apartment from the address solely and was very satisfied, particularly since it solved one more important problem - predicting the outcome in case of a new predictor level (since the predictor becomes continuous rather than categorical, you will be able to predict the price of delivery even there was no delivery for a given ZIP). Beautiful heatmaps is another bonus.
Here is a nice lecture (with a nice Italian accent) by Fabio Veronesi with examples of kriging with R.
http://www.fabioveronesi.net/r-course/lesson4.html
One problem that you will face is geocoding the GPS coordinates, and you can use the function geocode from ggmap package in R (using free service from Google up to some limit of queries per day, I queried for a week to get all done).
http://cran.r-project.org/web/packages/ggmap/ggmap.pdf
Hopefully, this was useful, happy predicting.
A: Neural networks have been used with well over 1000 categories (see the Google paper on image recognition). But yes, it is not easy, and you may need much much more data to learn.
Have you considered aggregating categories into larger groups instead?
