2
$\begingroup$

I am using 60 obseravation*90features data (all continuous variables) and the response variable is also continuous.
These 90 features are highly correlated and some of them might be redundant.

I am using gain feature importance in python(xgb.feature_importances_), that sums up 1.
I run xgboost 100 times and select features based on the rank of mean variable importance in 100 runs.

Let's say I choose the top 8 features and then, again run xgboost with the same hyperparameters on these 8 features, surprisingly the most important feature (when we first run xgboost using all 90 features) becomes least important when we run xgboost using top 8 variables.

Any feasible explanation for this?

Improve this question
$\endgroup$
6
  • $\begingroup$ Instead of the mean, look at the max and min rank of a feature. The top 10 features are always outperforming the others in terms of information gain or just on average? $\endgroup$
    – Laksan Nathan
    Dec 16, 2019 at 8:25
  • $\begingroup$ These top 10 features are almost outperforming other features in each iteration. I am running both the experiments (using 90 features or using 10 most important features ) 100 times. $\endgroup$
    – kamal kumawat
    Dec 16, 2019 at 8:42
  • $\begingroup$ …even with highly correlated/redundant information the most important feature is unlikely to be ranked last for 100 runs when using top 10 features. Are you using feature sampling (by level or tree) in XGBoost? You could try changing that parameter to see what changes. $\endgroup$
    – Laksan Nathan
    Dec 16, 2019 at 8:50
  • 1
    $\begingroup$ Other than that, I think with 60 observations for 90 features you are likely to have high variance in your ranking. $\endgroup$
    – Laksan Nathan
    Dec 16, 2019 at 8:51
  • $\begingroup$ Yes, I am using feature sampling by columns(0.8). To reduce high variance in the ranking, do you suggest higher iteration like 1000 or more? $\endgroup$
    – kamal kumawat
    Dec 16, 2019 at 9:05

1 Answer 1

Reset to default
3
$\begingroup$

The importance of correlated features shrinks in tree models. Intuitively, it is because two correlated features are somewhat equivalent in the information they bring, and therefore the tree can decide to split on any of the two. For this reason, two perfectly correlated features will split the total importance relative to the information they bring in two.

When you removed the other 82 features, these were (as you also said) highly correlated. What is very likely is that the one feature that was standing out in the 90 variables model was NOT correlated to any of the others (hence the high relative importance). Once you reduce the number of correlated variables, the ones that you have selected gather all the importance that before they were "splitting" with the other correlated features, ending up with more total importance than the other single variable which was uncorrelated from the rest.

I hope it makes sense. Try to look at the correlation matrix of the full variables, and hopefully you will see that that particular variable has less correlation dependencies than the other ones that ended up being more important than it.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Absolutely make sense but what I am really struggling to understand is that if some features are highly correlated then I am expecting their feature importance to be somewhat on the same level rather than one is very important and others correlated features relative importance close to zero if I am giving enough tress(>4000) and colsample(>.8). $\endgroup$
    – kamal kumawat
    Dec 17, 2019 at 7:22
  • 1
    $\begingroup$ This could be the correlation between the features 4-8 and features 9-12. After you removed the latter, the former features jump higher in the importance ranks. $\endgroup$
    – Fardin Abdi
    Jun 14 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.