5
$\begingroup$

Conceptually, a Decision Tree should work just fine with categorical variables.

Why does DecisionTreeClassifier require numeric variables in the training data set?

PS. I tried dummy variables (which ignore the mutual exclusivity) and weight-of-evidence (loses information), I wonder if there is a better approach...

Improve this question
$\endgroup$
2
  • $\begingroup$ also a very similar question here $\endgroup$
    – user3494047
    Oct 5, 2016 at 13:09
  • $\begingroup$ ok, definitely an answer which answers your question can be found here $\endgroup$
    – user3494047
    Oct 5, 2016 at 13:11

1 Answer 1

Reset to default
7
$\begingroup$

Decision Trees do work with categorical data. It says that Decision Trees are "Able to handle both numerical and categorical data." You just have to convert them to integers. For example {'A', 'B', 'C'} should be converted to the set {1,2,3}.

You can read more about the specific algorithm implemented in sci kit learn Tree algorithms section on the same page. Sci kit learn uses an optimized version of the CART algorithm. This algorithm makes a binary tree using thresholds on attributes which create the largest information gain. I am not sure as to the positives and negatives of this algorithm vs an algorithm that does create rule sets, since I am not the most familiar with the CART algorithm, but perhaps that is a different question.

EDIT: although I think the answer I suggested is very common, for example shown here and is the standard technique in libraries like WEKA, there is another way to map categorical features to ints/floats suggested here. In this answer it is suggested to add a set of features the same size as the number of values that the categorical feature can take (i.e. if 'red', 'green', and 'blue') and map them to some binary encoding of these features. For example 'red' would go to (1,0,0) 'green' would go to (0,1,0) and 'blue' would go to (0,0,1).

Note that not all algorithms for decision tree require giving numerical input values. There are decision tree algorithms (like the id3) which do not need numerical input values and treat features as actual categories. It depends on the implementation. It seems that for ease of implementation sci-kit learn has decided to use CART which treats everything as numeric.

Improve this answer
$\endgroup$
5
  • $\begingroup$ does the mapping matter? iow, if I map {'a','b','c'} to {2,3,1} instead of {1,2,3}, will I get an isomorphic tree? $\endgroup$
    – sds
    Oct 5, 2016 at 12:00
  • $\begingroup$ I believe you would get an isomorphic tree but I am having trouble finding the exact algorithm. From what I understand it creates (only) binary splits using a threshold (i.e. treating each feature as a continuous variable) which maximizes information gain. This seems like it should then, in the end, find an isomorphic tree although I have not proved or found a proof. using thresholds to create binary splits it is possible to get an isomorphic tree no matter what mapping you choose, the question is what the algorithm does.. $\endgroup$
    – user3494047
    Oct 5, 2016 at 12:30
  • $\begingroup$ It seems that if the integer variables are treated as ordinal (IOW, as if they were floats) rather than cardinal (IOW, unordered sets) the trees will not be isomorphic. $\endgroup$
    – sds
    Oct 5, 2016 at 12:42
  • $\begingroup$ I will try to prove that it is always possible to create an isomorphic tree using any mapping of cardinal values to a set of floats. It is a bit tedious, but intuitively seems definitely true to me. $\endgroup$
    – user3494047
    Oct 5, 2016 at 12:55
  • $\begingroup$ my question is more practical: does CART create isomorphic trees? But I got your point, thanks! $\endgroup$
    – sds
    Oct 5, 2016 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.