Feature importance in gradient boosted trees

Question

I am tuning the parameters of a gradient boosting regression tree algorithm and find it hard to understand the importance of some variables. Here is the case.. when the number of estimators is relatively low (let's say up to 250) two features that are highly correlated with the target are the most important ones. When I increase the number of trees, the two most important features are the ones that are not significantly correlated with the target.

Does it make sense for the algorithm to use those features as the most important ones? Thanks in advance.

Answer 1

It would make sense to have variable importance change with using different features and over boosting iterations. Said simply: a) combinations of weak features might outperform single strong features, and b) boosting will change its focus during iterations$^1$, so I could imagine the importance to change with different amount of iterations as well as different combinations of features.

$^1$ (using classification for the example): boosting assigns a weight to each sample which determines the samples importance for the modelling. If a sample is classified correctly the weight gets decreased, if it's classified wrong it gets increased. So boosting always "focuses on the part of data that is currently hard to classify" - hence, over iterations, this focus will change. In combination with different sets of features I could imagine a focus change to samples where those 2 features you mentioned become less important, compared to a combination of other features. Therefore other, more important features could show up in your importance statistics. But this will also depend on how exactly the average variable importance is calculated from the resulting ensemble of models.