I would like to know if someone knows of a way to group the number of levels of a feature that has 100's (even 1000's) of levels to a smaller number of levels - also, what number levels it should reduce to e.g. should it reduce from 200 levels to 10, 15 or 20?
I think what the OP is asking is whether or not you 'retain the same information' (or most of it) if you reduce the number of levels of a factor, and how to code such a thing. But let me back up.
Your statement: "With PCA you can reduce the number of features from 5000 to 10 and maintain a very similar accuracy between including all 5000 features or just taking the top 10 PCA features." is not necessarily correct. This depends on the data itself and how correlated your features are. If there are many highly correlated features, you may be able to retain much of the variance using a small set of principal components, but in other cases, this will not be true.
Now, whether or not you can restructure a factor, making it less granular, without losing any information is an empirical question. For instance, suppose you had 100 levels of a factor indicating 'degree of dislike for vanilla ice cream'. This may very well be too granular, in which case you might find that cutting it down to 4 does just as good a job. But again, it depends on the data.
If your question is how to accomplish such a task (I'll assume in R), there are several solutions. Here is one. Further assuming that you have an ordered factor:
#create a factor with 300 levels dat = data.frame('Class' = 1:300) dat$Class = factor(dat$Class) #assuming an ordered factor, convert to numeric then use cut to reduce to 10 levels dat$Class2 = cut(as.numeric(dat$Class),10,labels = FALSE)
Now, if your factor levels are not ordered (your post suggests it is not -- similar to a 'postal code'?), the above won't do what you want. You'll need to come up with a scheme to recode these variables at a higher level. I'm no expert in the postal sciences, but continuing with that example, I might consider using just the first 3 numbers.
It is hard to say how to group the levels based on the information you provided.
It can depend on how much data you have. Suppose you have $1000$ data points, then you may want to group into $10$ levels from $100$ levels. However, if you have $1,000,000$ data points, may be you want to group into $100$ levels. The underline idea is more levels would increase the "complexity" of the model. For large amount of data, you are less likely to have over fitting problem. So you can have more levels.
It can depend on how much information in original feature. Suppose, the original feature has many unique values. And most data would have few values. (80-20 rules, and most real world data like this). You may want to group all the infrequent levels into "Others".
It can depend on how this feature is related to the prediction target. Certain levels / values may have very strong correlation with the label, so you may group those levels specially.
It can depend on domain knowledge and real world requirement. All above are "data driven", where you look at data and decide how to group. On the other hand, in real world, we may have some constrains from the domain you are working on. For example you want do some differentiated advertising, but your marketing department only wants at most 5 groups instead of 20. Then you need to try your best under such constraint.