2
$\begingroup$

I already asked this question is SO; however, I realized that this may be a better place for this type of question.

I am well aware that when using categorical features with tree based models such as random forest and gradient boosting there is no need to drop one level from N-level categorical features. For example, the following color feature with three levels can be made three binary features.

Color|| Color_R | Color_B | Color_G
____ ||_________|_________|________
 R   ||   1     |   0     |   0
 B   ||   0     |   1     |   0
 G   ||   0     |   0     |   1

However, what about binary feature (E.g., TRUE/FALSE, MALE/FEMALE)? Should it be kept as a single binary feature (Option I below) or should it also be one-hot encoded into two binary features (Option II below)

Option I

Gender || Gender  | 
____   ||_________|
M      ||   1     | 
F      ||   0     | 
M      ||   1     | 

Option II

Gender || Gender_M | Gender_F 
____   || _________|_________
M      ||    1     |   0     
F      ||    0     |   1     
M      ||    1     |   0  
$\endgroup$
  • $\begingroup$ To whom it may concern -- I don't know why this is accumulating close votes. It seems perfectly on-topic to me. I think my answer shows why that's true: it focuses on statistical concepts, not how to use software. $\endgroup$ – Sycorax says Reinstate Monica Dec 4 at 0:10
2
$\begingroup$

It's true that you're not missing information when you use only $k-1$ categories. In linear models, we are all familiar with the dummy variable trap and the relationship between a model with $k-1$ levels and an intercept and a model with $k$ levels and no intercept. However, you're using a tree-based model, so the mechanics of how recursive binary splits work are important!

In the case of a factor with 2 levels, e.g. "red" and "blue", it's obvious that using the $k-1$ 1hot method is equivalent to choosing the $k$ 1-hot method. This is because NOT blue implies red. In this case, there is no difference.

But for $k$ categories, you'll need $k-1$ binary splits to isolate the the omitted level (the $k$th level). So if you have 3 levels, e.g. "red", "green", "blue", but you only include 1-hot features for "red" and "green", it will take 2 successive splits to isolate the "blue" samples. This is because if you split on "red", the children are nodes for red and NOT red = green OR blue. To isolate "blue" when the category "blue" is omitted from the coding scheme, you'll have to split again on "green" because then the children nodes of green OR blue will be blue and green.

As $k$ increases, this problem becomes more pronounced, as you'll require more splits. This may interact with your other hyperparameters in strange ways, because specifying a maximum tree depth is a common strategy to avoid overfitting with boosted trees/xgboost.

If isolating category $k$ isn't important, then this effect may not matter at all for your problem. But if category $k$ is important, you'll tend to grow very deep trees to try and isolate it, either via the categorical variables or else by identifying latent interactions of other variables.

$\endgroup$
0
$\begingroup$

Go with your Option I - there is no need to do one-hot encoding when there are only two categories.
These two columns Gender_M and Gender_F carry the exact same information (since it's binary, at least in your example).
I think some frameworks need binary classes to be one-hot encoded, but not features.

$\endgroup$
  • $\begingroup$ Itamar, Thank you for taking the time to answer my question. Just to be clear, what you are suggestion for non-linear models (e.g., DT) is that for categorical feature with k levels where k>2 ( e.g., the color feature where k=3) then create k features. But when k=2 (e.g., Gender feature, k=2) then use k-1 levels. Correct? But what will happen if the binary feature is missing? In this case a value of 0 doesn't indicate missing value the same as if a values of [0,0,0] would have indicated a missing value for the color feature. $\endgroup$ – thereandhere1 Dec 2 at 14:45
0
$\begingroup$

Welcome to the site.

Adding a second variable adds no information that is not already contained in the first, it is perfectly colinear.

Your example about color is redundant too: If an observation is neither of the first two classes, then it must be the third. While not every method suffers to the same extent from the dummy variable trap, adding a $k^{\text{th}}$ variable never adds information that isn't already contained in the first $k - 1$. This extends to any number of categories and isn't unique to binary variables.

Hence, you should go with option I.

$\endgroup$
  • $\begingroup$ Thank you for taking the time to answer my question and welcome message. I am well aware of the dummy trap , which I always assumed it is only an issue for linear models and not for non linear models such as DTs (am I correct ?) Let me repeat the follow up question I asked Itamar. Should I always drop one level no matter how many levels the feature have ( both for color and gender) or should I only drop one level when I have a binary attribute (i.e. gender ) and not for color? Isn't option II handles better cases where the feature is missing or have a new level which was not used for training? $\endgroup$ – thereandhere1 Dec 3 at 4:42
  • 1
    $\begingroup$ Missing is not the same as not being a a certain class. If you do what you propose, you are saying that there are three classes: Male, female and missing. While this is one way to explicitly model missingness, how to deal with missingness is in general is problem dependent and a different issue then how to encode a categorical variable. (For example, you could also just include a dummy variable for missingness itself and/or use (multiple) imputation to fill in the missing category.) $\endgroup$ – Frans Rodenburg Dec 3 at 4:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.