How XGBoost chooses between two features that gives the same information?

Question

I have 10 variables in a dataset (X1, X2, .., X10) plus the binary target variable (Y). When I train a model with xgboost.sklearn.XGBClassifier() and then plot the feature importances with xgboost.plot_importance(my_model, importance_type='gain), I get that X1 variable is the most important (~70%) for this first model. However, if I drop the X1 variable, train again, I get the X2 variable is the most important (~70%) in this second model with no significant differences in the final performance of both models. So, I compute the linear correlation between those two hoping it's high and I just get -22% (and this -22% makes business sense, so no problem with that)

My question

What criteria XGBoost uses to choose X1 over X2 when are both variables present? (And it gives a feature importance to X1 near 70% and to X2 near 0%), does this suggest that XGBoost already gets all the information of X2 using X1? If yes, how?

Answer 1

Yes, this is what $0\%$ importance for X2 in the presence of X1 suggests.

Now, the "how" is somewhat open-ended but the basic "how" is that the inclusion of X1 allows any interactions of X2 with other variables as well as the variation of (a discretised) X2 itself to be adequately captured in (a discretised form of) X1. Thus, X1 simply gives better splits than X2 so X1 is (almost) always picked instead of X2. The thread How does xgboost select which feature to split on? gives a further discussion on this point but the point to emphasise it that we do a greedy search and pick the optimal split.

Side-note: The "-22% correlation" actually might be very strong in certain domains, it might even be the case that the correlation is not higher simply because X2 is "noisy" (and thus once more we should favour X1). To that extent, when using tree-based learners, we use the ordering of the feature variable so linear correlation (Pearson) is less informative than rank correlation (Spearman) regarding the information presented to the learner.

All in all, yes, it is a bit unusual to have such a extreme behaviour (70% going to 0% in terms of importance) but if the improvements in our metric of including X2 are always outperformed by including X1 and X1 interacts similarly with X2 with the other feature variables, and those X2:Xn interactions are not informative towards our end goal than X1:Xn sure... it can happen. :)