A decision tree, while performing recursive binary splitting, selects an independent variable (say $X_j$) and a threshold (say $t$) such that the predictor space is split into regions {$X|X_j < t$} and {$X|X_j >= t$}, and which leads to greatest reduction in cost function.

Now let us suppose that we have a variable with categorical values in {$X$}. Suppose we have label-encoded it and its values are in the range 0 to 9 (10 categories).

  1. If DT splits a node with the above algorithm and treat those 10 values are true numeric values, will it not lead to wrong/misinterpreted splits?
  2. Should it rather perform the split based on == and != for this variable? But then, how will the algorithm know that it is a categorical feature?
  3. Also, will one-hot encoded values make more sense in this case?

The fundamental question is: what is the nature of the variable? There are two options:

  1. Variable is numeric, i.e., labeled as 0 to 9 where the magnitude of the number matters. An example is a rating.
  2. Variable is categorical. i.e., labeled as 0 to 9 where the magnitude of the number does not matter. An example is color (where 0 can stand for green, 1 can stand for blue, etc.)

Based on the wording of your question, I assume the nature of the variable is truly categorical (e.g., a color). With that assumption, here are the answers to your three questions:

  1. The decision tree may yield misinterpreted splits. Let's imagine a split is made at 3.5, then all colors labeled as 0, 1, 2, and 3 will be placed on one side of the tree and all the other colors are placed on the other side of the tree. This is not desirable.
  2. In a programming language like R, you can force a variable with numbers to be categorical. If you do that, then you won't run into the issues you are worried about. Here's an example using the mtcars dataset to convert a numeric variable to categorical:

#carb variable starts out as numeric

#convert carb variable to categorical
mtcars$carb_categorical <- as.factor(mtcars$carb)
  1. Yes, one-hot encoded values will make more sense.
  • $\begingroup$ Thank you for the detailed response! With respect to #2, does R implementation of DT check the values of categorical variables for equality rather than less and greater than values? (I have no experience in R, hence asking). $\endgroup$ Aug 9 '19 at 7:40
  • $\begingroup$ It does; the decision tree will treat the categorical variable as one-hot encoded values implicitly. Note that in R, factor variables are limited to a maximum number of levels. $\endgroup$
    – mnmn
    Aug 9 '19 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.