7
$\begingroup$

I set up an experiment with these 8 Regressor Methods:

sklearn package

  • DecisionTreeRegressor, RandomForestRegressor, ExtraTreesRegressor, GradientBoostingRegressor

other packages

  • CatBoostRegressor , ExplainableBoostingRegressor , LGBMRegressor, XGBRegressor

Every method ran with:

  • 2000 random_seeds and
  • 3 collinear features, where feature 2 and 3 are copies of the first one.

I measured a method's management of multicollinearity by shapley_values (except for the EBM since its feature importance is already comparable to shapley values)

In general, I discovered that dealing with multicollinearity depends on the methods implementation. By default not every Gradient Boosting Technique deals with it the same way. So my original answer on a similar topic was wrong.

Here is an Example Output with random seeds and a shapley_contribution for all three features. enter image description here

Now, after the experiment I feel unsatisfied with the results. So I want to improve the experiment:

  • Do you have other metrices or extensions in mind to see how the methods deal with multicollinearity compared to each other? e.g. plotting also the constant?
  • Do you know of a plot which could combine 8 example plots in one? So that people do not have to look at 8 graphics but instead at only one graphic?

If you want to dive deeper into this experiment, I listed every output plot, the notebook and the data here: Experiment files: https://github.com/OldPatrick/crossvalidated

Improve this question
$\endgroup$
3
  • $\begingroup$ Can you please edit this and make it more concise? 800+ words, with graphs and multiple questions is probably a sign that this needs more focus... $\endgroup$
    – usεr11852
    Nov 9, 2021 at 14:10
  • $\begingroup$ Ok I will do so tomorrow $\endgroup$
    – Patrick Bormann
    Nov 9, 2021 at 21:37
  • 2
    $\begingroup$ Ok, I reworked the whole question and focused on the things I'm interested in the most. $\endgroup$
    – Patrick Bormann
    Nov 10, 2021 at 19:13

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.