I would like to know if someone knows of a way to group the number of levels of a feature that has 100's (even 1000's) of levels to a smaller number of levels - also, what number levels it should reduce to e.g. should it reduce from 200 levels to 10, 15 or 20?

  • 4
    $\begingroup$ This is not a well structured question. Are you trying to group levels of a factor, or discretize a continuous variable? $\endgroup$ – HEITZ Jun 28 '16 at 8:12
  • $\begingroup$ I'm sorry if the question was not clear, I would like to group the number levels of a factor, hence reduce a factor with 100 levels to let's say 10 levels that will explain as much (or almost as much) variance as the original 100 levels would explain. $\endgroup$ – Charl Francois Marais Jun 28 '16 at 8:45
  • $\begingroup$ Presumably, then, this "factor" is an explanatory variable in a regression. But what does it represent? Is there some inherent order or topology to the levels of this factor? What is the purpose of the regression? Why do you want to group the factor? $\endgroup$ – whuber Jun 28 '16 at 16:35
  • 1
    $\begingroup$ Okay the goal is to simplify the model, if possible, and without losing a lot of predictive power. Another reason is that certain packages in R cannot handle factors with let's say 3k levels. The purpose doesn't matter, it is a general question-whether it is a count model or a binary classification model, having 3k levels will be a problem. No order and no topology, I'm asking a general question-imagine it as geographic indicators (a bit more granular than postal code). Also, what would the difference/impact be if it was ordered-I would guess then it is simpler, since you can group neighbors? $\endgroup$ – Charl Francois Marais Jun 28 '16 at 17:03
  • 1
    $\begingroup$ Answers that are much more useful and penetrating will become available if you would disclose any of the information I asked for. You might learn about splines, spatial regression models, GAMs, geostatistics, or any other number of techniques that way. A general question can receive only general answers, whose usefulness might be questionable or limited in any particular application. That's one reason we ask people to post questions that focus on their problem, rather than posting general questions. $\endgroup$ – whuber Mar 1 '17 at 14:56

I think what the OP is asking is whether or not you 'retain the same information' (or most of it) if you reduce the number of levels of a factor, and how to code such a thing. But let me back up.

Your statement: "With PCA you can reduce the number of features from 5000 to 10 and maintain a very similar accuracy between including all 5000 features or just taking the top 10 PCA features." is not necessarily correct. This depends on the data itself and how correlated your features are. If there are many highly correlated features, you may be able to retain much of the variance using a small set of principal components, but in other cases, this will not be true.

Now, whether or not you can restructure a factor, making it less granular, without losing any information is an empirical question. For instance, suppose you had 100 levels of a factor indicating 'degree of dislike for vanilla ice cream'. This may very well be too granular, in which case you might find that cutting it down to 4 does just as good a job. But again, it depends on the data.

If your question is how to accomplish such a task (I'll assume in R), there are several solutions. Here is one. Further assuming that you have an ordered factor:

#create a factor with 300 levels
dat = data.frame('Class' = 1:300)
dat$Class = factor(dat$Class)
#assuming an ordered factor, convert to numeric then use cut to reduce to 10 levels
dat$Class2 = cut(as.numeric(dat$Class),10,labels = FALSE)

Now, if your factor levels are not ordered (your post suggests it is not -- similar to a 'postal code'?), the above won't do what you want. You'll need to come up with a scheme to recode these variables at a higher level. I'm no expert in the postal sciences, but continuing with that example, I might consider using just the first 3 numbers.

| cite | improve this answer | |
  • $\begingroup$ Thanks for the answer, yes indeed I agree that this method would be applicable for ordered data - but this adds to my previous question (assuming for now that the data is ordered), how to determine the number of levels to reduce to? $\endgroup$ – Charl Francois Marais Jun 29 '16 at 5:59
  • $\begingroup$ Then regarding the postal code example, let's say you want to determine the like or dislike for vanilla ice cream based on region (postal code) then let's also say that the postal codes are completely random in it's structure (i.e Buffalo Grove, Chicago is AB-1,Beverly Hills, Los Angeles is JK-B-09) now if we have 10 000 records for each postal code then we have enough exposure. Working with this framework I would like to group the most similar postal codes and reduce let's say 3000 levels to 30 (or 40, 50, 60). Continued below... $\endgroup$ – Charl Francois Marais Jun 29 '16 at 6:07
  • $\begingroup$ The problem I can already think of with this is that if you include a second/third/fourth variable then the interaction of some of the levels of postal code might be significantly interact with other variables - and if you group the postal codes, some information might be lost in this process. So let's split this in 2 parts - 1 where there is only 1 variable (postal code) and then 2- multiple variables and handling this interaction. I really appreciate the help on this topic, for me it is really interesting and I am still learning a lot about Data Science and ML. :) $\endgroup$ – Charl Francois Marais Jun 29 '16 at 6:11
  • $\begingroup$ You must be from Chicago - no one knows Buffalo Grove otherwise. It sounds like you might want K-means clustering, but the clusters (however many you choose) will be based on the similarity (or dissimilarity) of the other features, not the feature you wish to recode. Also, the appropriate number of clusters is somewhat arbitrary. This may be a situation requiring extensive human intervention, if it cannot be solved algorithmically. If you really do have city names, you might parsing them with regex to get to a larger region, but without seeing your data I can't say. $\endgroup$ – HEITZ Jun 29 '16 at 6:27
  • $\begingroup$ :) I am actually not even from the states-I just know these suburbs.Just so that you know the question is really just in general what a person should do (or attempt).If I would apply kmeans, then I guess it would be just as good as removing postal code,and just use PCA on the region features?Getting a larger region might not be a good idea,since let's say for argument poor people like ice cream and rich people doesn't-then you really want region as granular as possible (even to street level maybe). But I agree it depends on what you are modelling (like large regions rain/no rain will be fine) $\endgroup$ – Charl Francois Marais Jun 29 '16 at 6:36

It is hard to say how to group the levels based on the information you provided.

  • It can depend on how much data you have. Suppose you have $1000$ data points, then you may want to group into $10$ levels from $100$ levels. However, if you have $1,000,000$ data points, may be you want to group into $100$ levels. The underline idea is more levels would increase the "complexity" of the model. For large amount of data, you are less likely to have over fitting problem. So you can have more levels.

  • It can depend on how much information in original feature. Suppose, the original feature has many unique values. And most data would have few values. (80-20 rules, and most real world data like this). You may want to group all the infrequent levels into "Others".

  • It can depend on how this feature is related to the prediction target. Certain levels / values may have very strong correlation with the label, so you may group those levels specially.

  • It can depend on domain knowledge and real world requirement. All above are "data driven", where you look at data and decide how to group. On the other hand, in real world, we may have some constrains from the domain you are working on. For example you want do some differentiated advertising, but your marketing department only wants at most 5 groups instead of 20. Then you need to try your best under such constraint.

| cite | improve this answer | |
  • $\begingroup$ Thanks for the response! I agree with what you said, but this is my general logic (I might be wrong - I am still learning). With PCA you can reduce the number of features from 5000 to 10 and maintain a very similar accuracy between including all 5000 features or just taking the top 10 PCA features. This can be done regardless if you have 5000 rows of data or 50 000 rows. So I was wondering if you could do something like PCA on the feature level. I do not know what you can do though. I don't know if this make sense either - but in my head it does :) $\endgroup$ – Charl Francois Marais Jun 28 '16 at 9:54
  • $\begingroup$ I am still revising my answer. But got confused by your comment, are you trying to process levels in one feature or trying to reduce many features? $\endgroup$ – Haitao Du Jun 28 '16 at 9:55
  • $\begingroup$ Reduce level in one feature (but possible do the same for other features with many levels as well) $\endgroup$ – Charl Francois Marais Jun 28 '16 at 9:57
  • $\begingroup$ that's what i thought. so if you reduce level in 1 feature, why PCA. And what's your suggestions on the 4 points I made? clear? $\endgroup$ – Haitao Du Jun 28 '16 at 10:00
  • $\begingroup$ Indeed not PCA, I meant that you can reduce many features to a few using PCA. Can you reduces many levels to a few using something else (not PCA, but something else) $\endgroup$ – Charl Francois Marais Jun 28 '16 at 10:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.