1
$\begingroup$

I have been going through notebooks available online where classification of mushrooms are done to poisonous(p) or edible(e) based on mushroom properties. The features present in the dataset are all categorical in nature. Have a look below to the sample data:

class   cap-shape   cap-surface cap-color   bruises odor    gill-attachment gill-spacing    gill-size   gill-color  ... stalk-surface-below-ring    stalk-color-above-ring  stalk-color-below-ring  veil-type   veil-color  ring-number ring-type   spore-print-color   population  habitat
0   p   x   s   n   t   p   f   c   n   k   ... s   w   w   p   w   o   p   k   s   u
1   e   x   s   y   t   a   f   c   b   k   ... s   w   w   p   w   o   p   n   n   g
2   e   b   s   w   t   l   f   c   b   n   ... s   w   w   p   w   o   p   n   n   m
3   p   x   y   w   t   p   f   c   n   n   ... s   w   w   p   w   o   p   k   s   u
4   e   x   s   g   f   n   f   w   b   k   ... s   w   w   p   w   o   e   n   a   g

I see the solutions available use LabelEncoder from scikit-learn to change this categorical properties to integers and then use them to DecisionTreeClassifier. While the accuracy after training has been quite good, what I don't get is the logic behind the classifier's split - since they are categorical in nature.

(Q1) Because, consider a split cap-color < yellow, does this make any sense? So is this approach fundamentally wrong?

(Q2) I can understand, we can change the categorical values to one-hot-encoded vectors and then train the DT classifier. Now if this classifier is providing you lesser accuracy, which one would one use in real-life?

One of the notebooks: here

$\endgroup$

2 Answers 2

0
$\begingroup$

You can certainly transform categorical labels to numbers and (mis-)treat them as you described. The magic lies within the configuration of the algorithm.

Lets have a look at DecisionTreeClassifer by sklearn which you mentioned reaching good results.

There are two criteria that enable a DecisionTree to still perform well even in the described setup, namely

max_depth int, default=None The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.

and

max_leaf_nodes int, default=None Grow a tree with max_leaf_nodes in best-first fashion. Best nodes are defined as relative reduction in impurity. If None then unlimited number of leaf nodes.

Given that both are by default of value None which is interpreted as unlimited, it allows the tree to split nodes until all leafs are pure.

Thus (A1) the split in so far makes sense as using numerical values enables you to use the existing sklearn implementation of the DecisionTree. It does not make much sense mathematically / or within reason of the domain, but it is an easy hack to get going.

$\endgroup$
0
$\begingroup$

Q1) It doesn't make much sense. Nonetheless, based on the problem and data, label encoder may help with the results, if the imposed order correlates with the target variable in some way. In principle, it's not too different than the mean encoding, where you can again put a single number for each category, and can compare them each other. My first choice would not be introducing an artificial order as label encoder does. Besides, a decision tree is capable of converging to a similar behavior with label encoded features as one-hot encoded ones, e.g. with the combination x < 3, and in the next branch x > 1, which will eventually mean x = 2 (or yellow) for one of the branches. However, in general, this is not an efficient way.

Q2) If the label encoded one gives better results, and you believe that this is statistically significant, then, it means either the imposed order correlates well with the target variable, and you've engineered a useful feature; or, the algorithm's capabilities do not get along well with one-hot encoded features. So, yes, you may use the model; or, you may want to try other algorithms.

$\endgroup$
2
  • $\begingroup$ your answer makes sense to me. Whatever encoding we may provide, the decision tree would come up with a different split ultimately classifying perfectly. What is troubling me is whether you can convince your stakeholders when you explain the model to them..? $\endgroup$
    – s510
    Feb 1, 2022 at 13:04
  • $\begingroup$ I'm not sure perfect classification is what you look for because it smells like overfitting. $\endgroup$
    – gunes
    Feb 1, 2022 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.