I'm working with some categorical data that I want to use for prediction with a machine learning algorithm. Many of my my categorical features have multiple categories that contain very few positive observations.

I've seen approaches that merge categories in scenarios such as this in order to create a category with more positive observations (where it makes sense conceptually) .

My question here is whether this is the best approach, and if so whether there is a rule of thumb for how small a category needs to be to justify merging its with another category? I'm also interested to know whether some kind of statistical analysis should be applied to categories before any decision is made to merge them, or whether it is legitimate in most cases to use a priori knowledge and common sense in merging categories!?

A relevant example of one of a category where I'm considering this approach is as follows:

Old Category    Size of category (% of all records)
Sunny           82%
Raining         17%
Snowing         0.1%
Foggy           0.9% 

New Category    Size of category (% of all records)
Good Weather    82%
Bad Weather     17%

Ideally, you would want to let the learning algorithm decide what features are important. Does it make sense to merge categories as you suggest? I don't know.

Important things to consider w.r.t. your problem:

  1. What do you actually want to predict? Do you want to (also) analyze the features?

  2. How big is your data? The percentage numbers have fewer meaning when you have many observations, 0.1% snowy days could mean that of your n=1000 observations you have only 1 snowy day or you observed 1000 snowy days of your n=1.000.000 sized sample.

  3. What is your learning algorithm? If n is very large and prediction accuracy is most important you might want to consider neural networks, which by themselves ideally will extract "features" from the features you present it.

  4. Be careful with "Common sense". It is often not the sense other people have. In your example, you merge sunny,rainy,foggy, and snowy into two very subjective categories "good" and "bad" weather. I think this is not a "good" idea, as no one but you can say if you put snowy as "good" weather or "bad" weather (I personally love snow, but I know others who hate it...)

  • $\begingroup$ Thanks for your input. In this instance the data set is quite large (500,000+ observations). I get the impression from your feedback that absolute numbers are more important than proportions here though? It's a binary classification problem, where assessing feature importance is as much of an goal as prediction. My plan is to use random forests, which I understand should be effective for both these aims! $\endgroup$
    – RDG
    Dec 17 '16 at 10:57
  • $\begingroup$ Yes, but these percentages are still nice to know (I changed "few meaning" to "fewer" meaning). But a powerful learner of course should learn more useful patterns from 1000 snowy days than from 1, regardless of how many other weather situations he observes. $\endgroup$
    – user142639
    Dec 17 '16 at 11:30

I would suggest plotting a histogram to see the class distribution. In the example below, our target is "death by age 75". We can see that over half of those with blood type 'b' are dead, yet there are relatively few observations having that blood type. We might want to define a minimum number of observations before we allow a unique class, e.g. require at least 30 observations - given that rule, we would bin class 'ab' and 'b' together. Imagining there were more classes, we could reduce further classes wherein the target is infrequent.

blood type to death indicator

Another suggestion is to use an optimal binning method: Use a machine learning algorithm, e.g. decision tree, to find a good combination of classes with respect to the target class.

  • $\begingroup$ Interesting. I am a bit confused by your response though. In the example you provided I would have assumed merging AB & B would be $\endgroup$
    – RDG
    Dec 17 '16 at 11:06
  • $\begingroup$ ...bad for predicting classification accuracy (given their seemingly opposite relationships with the target class)? $\endgroup$
    – RDG
    Dec 17 '16 at 11:07
  • $\begingroup$ the point is that you can't make much of decision about those classes because there aren't enough observations to make assumptions about the true distribution. $\endgroup$
    – dnbwise
    Dec 17 '16 at 19:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.