Are there examples where splitting on the best feature/threshold combination is not actually the best way to split the tree, and that better results could be got by choosing a different feature but with a lower gain at that time? Where it makes sense to defer the use of particular features until further down in the tree?


I think this could be the case if a feature exhibits either spurious correlation with the target, or the relationship with the target changes over time. In both of these cases splitting on the "best" feature can result in overfitting. The former can usually be fixed with more data; the latter is a problem for ML in general.

In practice, it may be worth trying feature subsampling, e.g. if you are using gradient boosted trees. This may prevent a few "good" features that don't end up generalizing well from dominating your trees. You could also try removing some of the most "important" features according to your model.

  • 1
    $\begingroup$ I'm not so concerned with overfitting actually. What's happening is that the most 'obvious' features are used in the first couple of splits, and then further down, when it starts to introduce new features, there are much fewer examples in the nodes => and I wonder is this impacting its ability to learn stuff beyond the most obvious? $\endgroup$ – housecat64 Aug 21 '18 at 6:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.