I want to train a classifier, say SVM, or random forest, or any other classifier. One of the features in the dataset is a categorical variable with 1000 levels. What is the best way to reduce the number of levels in this variable. In R there is a function called combine.levels() in the Hmisc package, which combines infrequent levels, but I was looking for other suggestions.

  • $\begingroup$ Is the categorical variable unordered? Approximately how many cases do you have? What is the frequency distribution across the categorical variable? $\endgroup$ – Jeromy Anglim May 6 '11 at 3:32
  • $\begingroup$ The levels are not ordered. I have around 10,000 observations. The frequency distribution is as follows: level A appears in around 11% of the observations. Level B appears in 8%. Level c appears in 5%. About 15 of these levels cover 50% of the observations in the dataset. $\endgroup$ – sabunime May 6 '11 at 14:56

How best to do this is going to vary tremendously depending on the task you're performing, so it's impossible to say what will be best in a task-independent way.

There are two easy things to try if your levels are ordinal:

  1. Bin them. E.g., 0 = (0 250), 1 = (251 500), etc. You may want to select the limits so each bin has an equal number of items.
  2. You can also take a log transform of the levels. This will squish the range down.

If the levels are not ordinal you can cluster the levels based on other features/variables in your dataset and substitute the cluster ids for the previous levels. There are as many ways to do this as there are clustering algorithms, so the field is wide open. As I read it, this is what combine.levels() is doing. You could do similarly using kmeans() or prcomp(). (You could/should subsequently train a classifier to predict the clusters for new datapoints.)

| cite | improve this answer | |
  • 3
    $\begingroup$ I don't know of a really good way to handle this other than to treat the categorical variable as a random effect. You can emulate that by using a quadratic (ridge) penalization process on the variable. My Regression Modeling Strategie book and course notes goes into this. $\endgroup$ – Frank Harrell May 3 '18 at 11:33
  • 1
    $\begingroup$ @FrankHarrell Two approaches come to mind: POlytomous variable Latent Class Analysis would be one (cran.r-project.org/web/packages/poLCA/poLCA.pdf), correspondence analysis another (e.g., statmethods.net/advstats/ca.html). $\endgroup$ – Mike Hunter May 3 '18 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.