Consider a split criterion for decision trees, which favors splits resulting in groups with as evenly distributed classes as possible. What will be the effect on the resulting decision trees compared to using the information gain criterion? How would trees generated with the suggested criterion be expected to perform in terms of accuracy on independent test examples?

The gain, $\Delta$, is a criterion that can be used to determine the goodness of a split:

$$ \Delta = I(\text{parent}) - \sum_{j=1}^k \frac{N(v_j)}{N} I(v_j) $$

where $I(\cdot)$ is the impurity measure of a given node, $N$ is the total number of records at the parent node, $k$ is the number of attribute values, and $N(v_j)$ is the number of records associated with the child node, $v_j$. When entropy is used as the impurity measure in the above equation, the difference in entropy is known as the information gain, $\Delta_{info}$.

$$ \text{Entropy}(t) = - \sum_{i=0}^{c-1} p(i|t) log_2 p(i|t) $$

where $c$ is the number of classes.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.