Here's my two cents. I can't say with full certainty, but, I guess, it very much depends on a model and data. If I understand this answerthis answer correctly, @gung advises to test your model(s) after dropping all and then some levels. However, the details on how exactly to perform the testing are rather fuzzy (at least, to me). Perhaps, he will be kind enough to expand on that for beginners like me.
You may also find relevant and useful this course notes document on logistic regression (in R
) by Professor Christopher Manning (Stanford University). Among other things, he describes dropping whole categorical variables (factors in R
terminology) and manipulations with categorical variable levels, such as collapsing several levels into a single one and other manipulations, as well as the impact of those actions on quality of regression models and interpretations of analysis results.