This is a generic question about xgboost. I ran the following code to see the important features of my data

imp <- xgb.importance(names(train),model = gbdt)

Weirdly, when I remove one of the most important features, my model improves (aka my ability to predict test data on Kaggle improves). Why would this be? How can you determine which features to leave in the data, and which to remove? Thanks

  • 1
    $\begingroup$ How are you measuring performance? Have you accounted for the standard error of the performance estimates? Is this related to the nature of your trian/test partition? Is there a non-deterministic component to your estimation procedure (such as row- or column-subsampling)? $\endgroup$
    – Sycorax
    Commented Mar 5, 2017 at 15:09

1 Answer 1


This will happen if the most important feature is highly correlated with some other features. It seems violate the tree based model features selection rules-it should immune to collinearity, but the colsample_bytree (colsample_bytree is a parameter in xgboost) is random select features, so it will select them both, but if you remove the redundant one, the result's changes is insignificant.

colsample_bytree: subsample ratio of columns when constructing each tree. Default: 1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.