5
$\begingroup$

Conceptually, a Decision Tree should work just fine with categorical variables.

Why does DecisionTreeClassifier require numeric variables in the training data set?

PS. I tried dummy variables (which ignore the mutual exclusivity) and weight-of-evidence (loses information), I wonder if there is a better approach...

$\endgroup$
2
  • $\begingroup$ also a very similar question here $\endgroup$ Commented Oct 5, 2016 at 13:09
  • $\begingroup$ ok, definitely an answer which answers your question can be found here $\endgroup$ Commented Oct 5, 2016 at 13:11

1 Answer 1

7
$\begingroup$

Decision Trees do work with categorical data. It says that Decision Trees are "Able to handle both numerical and categorical data." You just have to convert them to integers. For example {'A', 'B', 'C'} should be converted to the set {1,2,3}.

You can read more about the specific algorithm implemented in sci kit learn Tree algorithms section on the same page. Sci kit learn uses an optimized version of the CART algorithm. This algorithm makes a binary tree using thresholds on attributes which create the largest information gain. I am not sure as to the positives and negatives of this algorithm vs an algorithm that does create rule sets, since I am not the most familiar with the CART algorithm, but perhaps that is a different question.

EDIT: although I think the answer I suggested is very common, for example shown here and is the standard technique in libraries like WEKA, there is another way to map categorical features to ints/floats suggested here. In this answer it is suggested to add a set of features the same size as the number of values that the categorical feature can take (i.e. if 'red', 'green', and 'blue') and map them to some binary encoding of these features. For example 'red' would go to (1,0,0) 'green' would go to (0,1,0) and 'blue' would go to (0,0,1).

Note that not all algorithms for decision tree require giving numerical input values. There are decision tree algorithms (like the id3) which do not need numerical input values and treat features as actual categories. It depends on the implementation. It seems that for ease of implementation sci-kit learn has decided to use CART which treats everything as numeric.

$\endgroup$
5
  • $\begingroup$ does the mapping matter? iow, if I map {'a','b','c'} to {2,3,1} instead of {1,2,3}, will I get an isomorphic tree? $\endgroup$
    – sds
    Commented Oct 5, 2016 at 12:00
  • $\begingroup$ I believe you would get an isomorphic tree but I am having trouble finding the exact algorithm. From what I understand it creates (only) binary splits using a threshold (i.e. treating each feature as a continuous variable) which maximizes information gain. This seems like it should then, in the end, find an isomorphic tree although I have not proved or found a proof. using thresholds to create binary splits it is possible to get an isomorphic tree no matter what mapping you choose, the question is what the algorithm does.. $\endgroup$ Commented Oct 5, 2016 at 12:30
  • $\begingroup$ It seems that if the integer variables are treated as ordinal (IOW, as if they were floats) rather than cardinal (IOW, unordered sets) the trees will not be isomorphic. $\endgroup$
    – sds
    Commented Oct 5, 2016 at 12:42
  • $\begingroup$ I will try to prove that it is always possible to create an isomorphic tree using any mapping of cardinal values to a set of floats. It is a bit tedious, but intuitively seems definitely true to me. $\endgroup$ Commented Oct 5, 2016 at 12:55
  • $\begingroup$ my question is more practical: does CART create isomorphic trees? But I got your point, thanks! $\endgroup$
    – sds
    Commented Oct 5, 2016 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.