9
$\begingroup$

A question about multicollinearity for random forests has been asked and answered, but what about boosted trees?

$\endgroup$

1 Answer 1

3
$\begingroup$

I believe I can answer that, although it is an old one:

Boosted Trees are immune to multicollinearity: https://datascience.stackexchange.com/questions/12554/does-xgboost-handle-multicollinearity-by-itself

See also the newest implementation of Boosted Trees with EBM from Microsoft: https://interpret.ml/docs/ebm.html

The boosting procedure is carefully restricted to train on one feature at a time in round-robin fashion using a very low learning rate so that feature order does not matter. It round-robin cycles through features to mitigate the effects of co-linearity and to learn the best feature function for each feature to show how each feature contributes to the model’s prediction for the problem.

But! As you can see from the first link. The second answer there highlights, that boosted trees can not work out multicollinearity when it comes to inference or feature importance.

Boosted Trees do not know, if you for example have added a second feature which is just perfectly linearly dependent from another. The Trees will just say that both features (the original one and the artifical one) are now important maybe they will share the feature importance. just make a simple experiment on that. You will see they can not deal with multicoll. in terms of yeah lets say causality.

If you would want such a thing you first need to aggregate features or do a regularization method.

Update 2022/1/17

I made an experiment examining the explanatory part of the multicollinearity in boostes trees and gams and decision trees. While for prediction, multicollinearity has no effect, the explanatory part is highly influenced by it. So far, only the EBM offers a handling of multicollinearity due to its round robin procedure.

See my other post: How shap values behave in terms of multicollinearity in Trees, Ensemble, GradientBoosting and GAM/Boosting

$\endgroup$
2
  • $\begingroup$ The problem with multicollinearity is that in many applications it indicates future values might be very far from the training values (or typical values) in the sense of Mahalanobis distance. It's difficult to see how any procedure possibly could anticipate and correct for this, except perhaps for warning about the possibility. $\endgroup$
    – whuber
    Jan 17, 2022 at 15:38
  • $\begingroup$ Then you should probalbx read and test the EBm, I guess you are referring to the general behavior of time series. Where this point might be true, but multicollinearity always refers to a lot of minimas, where prediction of a future value may be the same, but the features contributing to add may vary. You can steer that with a implementation as I have shown in my experiments. So, I guess only saying a warning is everything we have might not be true. $\endgroup$ Jan 17, 2022 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.