I set up an experiment with these 8 Regressor Methods:

sklearn package

  • DecisionTreeRegressor, RandomForestRegressor, ExtraTreesRegressor, GradientBoostingRegressor

other packages

  • CatBoostRegressor , ExplainableBoostingRegressor , LGBMRegressor, XGBRegressor

Every method ran with:

  • 2000 random_seeds and
  • 3 collinear features, where feature 2 and 3 are copies of the first one.

I measured a method's management of multicollinearity by shapley_values (except for the EBM since its feature importance is already comparable to shapley values)

In general, I discovered that dealing with multicollinearity depends on the methods implementation. By default not every Gradient Boosting Technique deals with it the same way. So my original answer on a similar topic was wrong.

Here is an Example Output with random seeds and a shapley_contribution for all three features. enter image description here

Now, after the experiment I feel unsatisfied with the results. So I want to improve the experiment:

  • Do you have other metrices or extensions in mind to see how the methods deal with multicollinearity compared to each other? e.g. plotting also the constant?
  • Do you know of a plot which could combine 8 example plots in one? So that people do not have to look at 8 graphics but instead at only one graphic?

If you want to dive deeper into this experiment, I listed every output plot, the notebook and the data here: Experiment files: https://github.com/OldPatrick/crossvalidated

  • $\begingroup$ Can you please edit this and make it more concise? 800+ words, with graphs and multiple questions is probably a sign that this needs more focus... $\endgroup$
    – usεr11852
    Nov 9, 2021 at 14:10
  • $\begingroup$ Ok I will do so tomorrow $\endgroup$ Nov 9, 2021 at 21:37
  • 2
    $\begingroup$ Ok, I reworked the whole question and focused on the things I'm interested in the most. $\endgroup$ Nov 10, 2021 at 19:13


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.