If I plot the feature importance of my xgboost model I get for example f10,f3,f7,f99,... as the most important features.
Now I decided to remove f3 and I imagined the new feature importance would be f10,f7,f99,... but what happened is: f10,f18,f99,f50,...
Xgboost seems to choose an entire different approach or something.
Can someone clarify? How can f7,f99 be such important feature but if I remove f3 they become useless????
You should consider the interaction or combined effect between features, so is very reasonable to you results. But if the features is redundant (highly correlated with other features) , remove it will not affect the results.
XGBoost website answers your question with a good example, which is partially answered by jumboRumbo. As it says:
In boosting, when a specific link between feature and outcome have been learned by the algorithm, it will try to not refocus on it (in theory it is what happens, reality is not always that simple). Therefore, all the importance will be on feature A or on feature B (but not both). You will know that one feature have an important role in the link between the observations and the label. It is still up to you to search for the correlated features to the one detected as important if you need to know all of them.
So, as you remove one feature, you don't get to keep the exact order for feature importance again, unless the features were completely orthogonal (uncorrelated).
