2
$\begingroup$

I'm using Xgboost's gradient boosted tree model for binary classification. It has a max_depth parameter, which specifies the maximum depth of each individual tree. This makes me assume that it can potentially stop splitting and end with a tree that is shallower than the max_depth tree parameter.

When building a tree, what stopping condition does Xgboost use to decide when to stop growing the tree (when to stop splitting a particular node any further)? I checked the original papers and the documentation and couldn't find any description of this.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

A split is decided based on the loss it produces. You can see the exact form in equation (7) of XGBoost: A Scalable Tree Boosting System - Chen, Guestrin. Specifically, we look at the drop in loss by doing the split, but then subtract $\gamma$, which is a regularization hyperparameter. In other words if the loss drop exceeds $\gamma$, we honor the split. If not, the split isn't made. I would think then that max_depth is reached only if the sequence of splits exceed $\gamma$ in loss drop. Otherwise if $\gamma=0$ there doesn't seem to be anything preventing the algorithm from attaining max depth.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Is it really the gain in accuracy? Gradient boosting generally uses regression trees, which greedily minimize the mean squared error. $\endgroup$
    – Matthew Drury
    Commented Jun 2, 2017 at 18:27
  • 2
    $\begingroup$ @MatthewDrury: It's formally a lowering of the loss. I was thinking along the lines of classification. Corrected, thanks. $\endgroup$
    – Alex R.
    Commented Jun 2, 2017 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.