2
$\begingroup$

If a feature has already split, will it hardly be selected to split again in the subsequent tree in a Gradient Boosting Tree? It is motivated by the fact that for the heavy correlated features in a single tree, usually only one of them will be select to split as their uncertainty will remain few after a splitting. Now in Gradient Boosting Tree, is residual similar with the uncertainty?

Currently I happened to how heavy correlated feature affect the feature importance selected by Gradient Boosting Tree. I guess the result is that Gradient Boosting Tree will only select the importance from one of correlated features just like LASSO.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

If a feature has already split, there is a good chance that it will be selected again either within the same tree or in the next trees. The main reason is the following: A tree split is effectively a (recursive) partitioning of the sample space through piece-wise flat functions. Standard tree construction methodologies do not preclude selecting the same feature multiple time so if a particular feature is highly predictive it can be selected again. It has to been noted here that especially when the trees are strongly regularised (e.g. by having a relatively shallow depth) subsequent trees are very likely to reuse a feature $x$ if that is highly predictive.

Regarding the appearance of heavy correlated features: Similar to the case of the LASSO, this correlation leads to instability during variable selection. Assuming a feature $x$ and heavily correlated version $x_{\text{c}}$, there is no reason why features $x$ and $x_{\text{c}}$ won't be used almost interchangeably, given they will have similar cost reduction observed after splitting on each of them as single features. This inability to adequately distinguish between correlated features is actually one of the reason why some variable importance metrics like the number of times a feature is used for splitting or the feature's permutation importance suffer greatly from the occurrence of correlated features; in such instances individual feature importance is attenuated and usually "smeared" across the group of correlated features. This also due to the fact that in quite a few instances we select a subset of our available predictive features when constructing individual trees, as such if $x$ is predictive but not selected but $x_{\text{c}}$ is selected, then our learner will happily use $x_{\text{c}}$ instead of $x$.

So to recap: A given feature $x$ might be use multiple times within a tree and across trees. Highly correlated versions of $x$, $x_{\text{c}}$, are equally likely to be used as $x$ itself, thus making the individual importance of $x$ (and of $x_{\text{c}}$) downward biased. This bias being especially prominent in relation to a third feature $x_1$ that might be not as predictive as $x$ but does not have any sibling features being included in training set).

Improve this answer
$\endgroup$
6
  • $\begingroup$ Could you confirm my understanding, 1. if a feature is not highly predictive and it has already split, this feature will hardly be selected to split again in the subsequent tree in Boosting? 2. individual feature importance is attenuated and usually "smeared" across the group of correlated features. under such case, boosting is relative better than random forest (random forest may exactly take the average of importance, if the number of tree is large)? $\endgroup$
    – user6703592
    Commented Dec 18, 2021 at 12:37
  • $\begingroup$ For 1. If a feature is not highly predictive, I don't see why would it be picked to begin with (assuming we are not doing extremely randomised forests or something) so I don't wee we can really comment on that intelligently. For 2. both GBM and RF will suffer similarly in that case. Ultimately variable importance measures in GBM and RF are pretty similar, GBMs and RF aren't too dissimilar anyway, we can even thing a RF as an GBM trained under extremely large drop-out regularisation (DART booster) - which would absurdly wasteful to o but theoretically possible. $\endgroup$
    – usεr11852
    Commented Dec 18, 2021 at 12:59
  • $\begingroup$ For 1, I mean if a feature splits in some node of the first boosting tree (may last few nodes, not split at very beginning), will it hardly split in the second, third... boosting tree? For 2, could you explain more on a RF as an GBM trained under extremely large drop-out regularisation (DART booster), I am happy to know this new point. $\endgroup$
    – user6703592
    Commented Dec 18, 2021 at 13:22
  • $\begingroup$ For 1. No, it can still be used. In general as long as splitting on that feature allows for some reasonable cost reduction it is good to go. For 2. the canonical reference for DART trees is DART: Dropouts meet Multiple Additive Regression Trees, the XGBoost page on the matter is a bit easier to digest at first instance (the paper is not hard to follow though). $\endgroup$
    – usεr11852
    Commented Dec 18, 2021 at 13:27
  • 1
    $\begingroup$ I get your point, RF is the case that GBM dropout all the trees each round. $\endgroup$
    – user6703592
    Commented Dec 18, 2021 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.