I already asked this question is SO; however, I realized that this may be a better place for this type of question.

I am well aware that when using categorical features with tree based models such as random forest and gradient boosting there is no need to drop one level from N-level categorical features. For example, the following color feature with three levels can be made three binary features.

Color|| Color_R | Color_B | Color_G
____ ||_________|_________|________
 R   ||   1     |   0     |   0
 B   ||   0     |   1     |   0
 G   ||   0     |   0     |   1

However, what about binary feature (E.g., TRUE/FALSE, MALE/FEMALE)? Should it be kept as a single binary feature (Option I below) or should it also be one-hot encoded into two binary features (Option II below)

Option I

Gender || Gender  | 
____   ||_________|
M      ||   1     | 
F      ||   0     | 
M      ||   1     | 

Option II

Gender || Gender_M | Gender_F 
____   || _________|_________
M      ||    1     |   0     
F      ||    0     |   1     
M      ||    1     |   0  
  • $\begingroup$ To whom it may concern -- I don't know why this is accumulating close votes. It seems perfectly on-topic to me. I think my answer shows why that's true: it focuses on statistical concepts, not how to use software. $\endgroup$ – Sycorax says Reinstate Monica Dec 4 '19 at 0:10

It's true that you're not missing information when you use only $k-1$ categories. In linear models, we are all familiar with the dummy variable trap and the relationship between a model with $k-1$ levels and an intercept and a model with $k$ levels and no intercept. However, you're using a tree-based model, so the mechanics of how recursive binary splits work are important!

In the case of a factor with 2 levels, e.g. "red" and "blue", it's obvious that using the $k-1$ 1hot method is equivalent to choosing the $k$ 1-hot method. This is because NOT blue implies red. In this case, there is no difference.

But for $k$ categories, you'll need $k-1$ binary splits to isolate the the omitted level (the $k$th level). So if you have 3 levels, e.g. "red", "green", "blue", but you only include 1-hot features for "red" and "green", it will take 2 successive splits to isolate the "blue" samples. This is because if you split on "red", the children are nodes for red and NOT red = green OR blue. To isolate "blue" when the category "blue" is omitted from the coding scheme, you'll have to split again on "green" because then the children nodes of green OR blue will be blue and green.

As $k$ increases, this problem becomes more pronounced, as you'll require more splits. This may interact with your other hyperparameters in strange ways, because specifying a maximum tree depth is a common strategy to avoid overfitting with boosted trees/xgboost.

If isolating category $k$ isn't important, then this effect may not matter at all for your problem. But if category $k$ is important, you'll tend to grow very deep trees to try and isolate it, either via the categorical variables or else by identifying latent interactions of other variables.

  • 1
    $\begingroup$ (+1) I hadn't thought of the class isolation problem before! $\endgroup$ – Frans Rodenburg Dec 12 '19 at 4:37

Go with your Option I - there is no need to do one-hot encoding when there are only two categories.
These two columns Gender_M and Gender_F carry the exact same information (since it's binary, at least in your example).
I think some frameworks need binary classes to be one-hot encoded, but not features.

  • $\begingroup$ Itamar, Thank you for taking the time to answer my question. Just to be clear, what you are suggestion for non-linear models (e.g., DT) is that for categorical feature with k levels where k>2 ( e.g., the color feature where k=3) then create k features. But when k=2 (e.g., Gender feature, k=2) then use k-1 levels. Correct? But what will happen if the binary feature is missing? In this case a value of 0 doesn't indicate missing value the same as if a values of [0,0,0] would have indicated a missing value for the color feature. $\endgroup$ – thereandhere1 Dec 2 '19 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.