I have a dataset from which I built lags at different levels to use as features in the XGBoost model. When I ran XGBoost models on H2O, the model is picking up the features which contain higher values as the most important features. When I tried GBM model on the same dataset, it's picking the correct features which are more correlated to the prediction column as important features. Does anyone know why it's happening?

Eg: If I run XGBoost model on H2O on the below sample dataset to predict Y (categorical variable with levels 0,1), the model is giving A as the most important factor instead of B which is actually the most important factor according to GBM:


10 5 6 1 6 0

20 9 7 3 8 1

15 8 5 1 2 1

13 9 2 2 7 1

18 5 7 1 8 0

I normalised the columns and tried XGBoost and it too gave the wrong columns as the most important columns


There could be multiple reasons for why you might be seeing this behavior, but I think part of the solution would be to consider how XGBoost and GBM calculate variable importance and then see if the results make sense given each algorithms calculations. In addition, it would be good to check if you have correlated features and whether the most important features are correlated with the features you think should be the most important.

For example, for GBM variable importance is determined by calculating the relative influence of each variable: whether that variable was selected during splitting in the tree building process and how much the squared error (over all trees) improved (decreased) as a result.

It would also be helpful to know how many levels you have in each categorical column. In general, you need to be careful with high cardinality variables because GBM's can use them to memorize solutions rather than learn from the data, so it is always good to check whether the most important features in your variable importance table are high cardinality features.

Please take a look this notebook on best practices for handling categorical features and see if after reading this you have a better sense of why you are getting the results you are getting: https://github.com/h2oai/h2o-tutorials/tree/master/best-practices/categorical-predictors in particular take a look at this notebook https://github.com/h2oai/h2o-tutorials/blob/master/best-practices/categorical-predictors/gbm_drf.ipynb.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.