It's a pity for randomForest in R.

But, for instance, if we have a categorical predictior with 100 categories, there is a solution in which we create 100 predictors each having 2 categories (0,1) (dummy variables), replace the 100-category predictor by 100 predictors and build the model? Evidently it will not work. My questions are:

  • Why isn't it working?

  • Do we have the same problem in Python?

  • 2
    $\begingroup$ I don't like your starting sentence. It is a too broad statement, has little to do with your question and is backed up by "everybody knows" . $\endgroup$ – Soren Havelund Welling Oct 13 '15 at 9:46
  • $\begingroup$ in R, try extraTrees or Rborist instead which will test a sub sample of possible factorial splits. $\endgroup$ – Soren Havelund Welling Oct 13 '15 at 9:48

Python does not handle factor directly (see : this stack overflow thread for more details). You can either encode them (turn them into ints, enforcing an order and making some splits "hard to see" for the algorithm). It is legitimate to do this if you suspect a natural order of your factors.

Alternatively, you can "one-hot" encode them (creating dummy variables). If you create dummy variables for a factor with a large number of levels, a lot of columns will correspond to this specific value, giving it more importance (each tree will be grown learning from a several number of these columns).

Indeed, the trees are grown on a sub-sample of the columns (by default, the columns are chosen randomly with equal probability). If you have two numeric predictors and a factor with 100 levels, almost all your trees will not take the information about the numeric values into account.

R on the other hand, will try every partition of the levels of the factor. See How to choose the split in Random forest for categorical predictors (features)? for more information about this. Having un-encoded factors usually makes it much slower. If you had more than 53 levels, the number of splits to try is $2^{53}$ which would be irrealistically slow. And the number of split to consider doubles at every new level.

  • $\begingroup$ If we create dummy variables, I don't see why we are giving it more importance. $\endgroup$ – Metariat Oct 13 '15 at 16:21
  • $\begingroup$ @QuangDOXuan thanks for pointing this out, I detailed the why $\endgroup$ – RUser4512 Oct 13 '15 at 16:24
  • $\begingroup$ great, very helpful, we have the problem with the importance level of the predictors, but does it impact the performance of the model? If yes, in case there is no natural order of the categorical predictors, how Python handle it? $\endgroup$ – Metariat Oct 13 '15 at 16:31
  • $\begingroup$ In most of the cases, it harms performance. sklearn's randomForest expects you to provide a numeric matrix, I don't think you can make it can handle factors the way R does it... $\endgroup$ – RUser4512 Oct 13 '15 at 16:35
  • $\begingroup$ If I understood correctly, what you are saying is that: Python is has no problem with the predictors having a natural order, but Python is not very efficient with the predictors without natural order? Sorry I didn't know very much about the implementation of random forests in Python $\endgroup$ – Metariat Oct 13 '15 at 19:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.