2
$\begingroup$

With respect to growing a classification tree: Does growing with Gini or Cross-entropy (CE) imply we would grow the tree until every leaf is pure (in case of no other stopping criteria)? Put differently: is Gini/CE always benefitted (reduced) by an additional split if the nodes are not completely pure yet?

And in contrast, is there for the misclassification error such a situation, where nodes are not pure yet but we don't continue growing as a further split would not reduce the misclassification error?

$\endgroup$

1 Answer 1

2
$\begingroup$

Suppose your model has 1 feature. You've been constructing a tree and you've reached a node where you have 6 samples (3 positives and 3 negatives) in your training data. In this scenario, for these particular samples, the feature values are identical. Any split you choose on your feature will put the samples in the same child node, and the purity of that child node will be the same as the parent node.

$\endgroup$
3
  • $\begingroup$ Very nice! I haven't thought about the scenario of having at some point equivalent observations at all. But if this is not the case, Gini and CE would still split while missclassification error might not split if both resulting child nodes would have the same class right? $\endgroup$
    – J3lackkyy
    Jul 27, 2021 at 18:28
  • $\begingroup$ Put differently: if we have a sample where the observations are distinct different from each other, would Gini/CE continue growing if the nodes are not pure yet? $\endgroup$
    – J3lackkyy
    Jul 27, 2021 at 18:30
  • 1
    $\begingroup$ @J3lackkyy Yes. To see why, write down a small number of observations with distinct feature values, and carry out the steps of the algorithm you have in mind. The main caveat here is that some algorithms (e.g. sklearn's Random Forest) give fine control over some "early stopping" criteria: minimum node size, minimum information gain to split, etc. $\endgroup$
    – Sycorax
    Jul 28, 2021 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.