1
$\begingroup$
  1. Suppose I have a mixture of some categorical features and numerically continuous features. I would like to train a classifier based on the features by RandomForestClassifier() in SciKi Learn.

    Random forest is said to handle both categorical features and numerically continuous features.

    But in my dataset, the categorical features' values are represented as numbers, such as id numbers, 1000, 1003, .... Will that be a problem to the random forest learning algorithm?

  2. Also does standardizing each feature so that its sample mean is zero and sample variance is one, make sense to categorical features which are faked as numbers, when using random forests? I saw someone did this, and I can't figure out if that make sense.

Thanks.

$\endgroup$
2
$\begingroup$
  1. Yes; however, it is problematic not for the learning algorithm per se but for interpretation: it assumes directional continuity (eg sample id 1500 is 50% 'greater' than sample id 1000), thereby becoming a meaningless pseudo-continuous variable. I tried to simulate this and it mattered for both IncMSE and NodePurity rankings. How much it would matter though would depend on your particular dataset.
  2. This seems a) unnecessary for random forests and b) incorrect for reasons specified above.
$\endgroup$
  • $\begingroup$ Thanks. How shall we specify faked numerical and real categorical features to RandomForestClassifier()? $\endgroup$ – Tim May 14 '15 at 20:45
  • $\begingroup$ @Tim: I think Dougal answered this question; I use RandomForests in R not Scikit, but imho it may be problematic if the latter does not explicitly support categorical predictors, because ability to handle several classes of predictors is one of the advantages of RF. Often considerable follow-up tuning involves handling the ratios of categories for categorical predictors. $\endgroup$ – katya May 14 '15 at 21:47
2
$\begingroup$

Scikit-learn doesn't directly support for categorical features in trees or forests (see comment by one of the core developers on SO, pointing to a currently stagnant implementation attempt).

If you leave them coded as id numbers, this imposes an ordering on the classes, and the forest will ask questions that rely on that ordering and are probably quite unnatural. Because the inputs are treated as real numbers, it'll make splits only like class_id <= 1004, which makes sense only if the classes actually have a natural order to them. This is actually problematic for the model's expressivity: it's possible to ask first class_id <= 1004 and then class_id > 1003 to pick out a single class, but the greedy method used to learn trees will often make the first split look not very good and so bias against this kind of question.

The typical workaround is to use a one-hot encoding, so that each categorical feature can be considered independently. This messes with the sampling procedure in the forests a bit (a given tree may be able to ask about class id 1000 but not 1003), but it's much better than the alternative.

I agree with katya that standardizing is unnecessary for random forests. Each node of the tree will consider split points according to some kind of purity measure after the split (in scikit-learn, either entropy or Gini coefficient). If you standardize the variables, you just change the scales of the variables monotonically, which means each possible split in the original space has an exactly equivalent split in the standardized space. It's possible there's some kind of regularization used in forests sometimes that's sensitive to scaling, but as far as I know it should end up with exactly equivalent models.

$\endgroup$
  • $\begingroup$ Thanks. why standardizing is unnecessary for random forests? Does it make any difference? $\endgroup$ – Tim May 14 '15 at 20:59
  • $\begingroup$ @Tim edited in an explanation. $\endgroup$ – Dougal May 14 '15 at 21:04
  • $\begingroup$ Also, note for your code that OneHotEncoder assumes the ids are $0, 1, \dots, n_\text{classes}$. You can use LabelEncoder to easily convert your current ids to that form. $\endgroup$ – Dougal May 14 '15 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.