2
$\begingroup$

I have a model with about 200,000 training observations, where I am regressing, with 4 factors and 2 continuous variables. One of my features has 927 levels, which is causing the R implementation of randomForest to fail (it has a limit of 32 levels for any feature). Unfortunately, I don't see a simple way to avoid using this factor, or to decompose it into a series of continuous variables. Since my predictors are a mix of categorical and continuous, I thought of trees. Can anyone suggest a different implementation (package or language), ML approach, or a better way to pre-process or massage my inputs?

$\endgroup$
2
$\begingroup$

I'm not an R user, but some general comments about random forest

Discrete features can be treated two ways depending on their property

  1. if they have a clear ordering, eg year, (bad/med/good) result, then you can just treat them as continuous.
  2. if they can't, eg gender,company name, then they should be dummy'd.
$\endgroup$
  • $\begingroup$ My factors fall into the second category. When I dummy them, I find myself with roughly 2000 variables, and only a few continuous variables. I can run a linear or generalized linear model on this system. Are there other approaches you recommend? $\endgroup$ – Ed Fine Mar 17 '13 at 16:12
1
$\begingroup$

You might try representing that one column differently. You could represent the same data as a sparse dataframe with dummy variables.

Minimum viable code;

example <- as.data.frame(c("A", "A", "B", "F", "C", "G", "C", "D", "E", "F"))
names(example) <- "strcol"

for(level in unique(example$strcol)){
      example[paste("dummy", level, sep = "_")] <- ifelse(example$strcol == level,     1, 0)
}
$\endgroup$
1
$\begingroup$

Have you tried cforest in party package? I know it can handle more than 30-40 levels but I am not sure about 900 levels.

http://www.inside-r.org/packages/cran/party/docs/cforest

$\endgroup$
0
$\begingroup$

You can use sklearn (python) and use OneHotEncoding.

OneHotEncoder basically creates one column per each level for a categorical variable (927 in your case). There created columns can take only 0 or 1 as values. This method by definition puts no limit on the number of levels. However, it does not capture the interaction between levels the same way as R does. The reason cforest doesn't allow you to have more than 30 levels is the huge number of permutations (~2^30).

Sklearn aside, I suggest finding a way to reduce your number of levels.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.