Combining XGBoost and LightGBM

Question

I'm working on a text classification problem and I am comparing LightGBM and XGBoost performances. Both on train and test sets I get roughly the same accuracy metrics, but what looks amusing to me is that feature importances (as well as shapley values and permutation importances) look quite different (some of the most important features for one algorithm are never used by the other and viceversa).

Of course there may be some collinearity between my variables, but I don't get how one may choose one variable over another.

And here's my question: is there any way of soundly combining the two? I don't have that many data to stack them btw. By combining I mean anything from feature selection to prediction combining.

Thank you

Answer 1

In the absence of enough data to perform stacking, just simple model averaging might be adequate for predictions. It will (typically) offer some improvement in terms of prediction errors. Dormann et al. (2018) "Model averaging in ecology: a review of Bayesian, information-theoretic and tactical approaches for predictive inference" offers a great and accessible overview of the subject.

In terms of feature selection, I would push back and ask what is the reason behind it. Unless we have some particular constraints (e.g. cost of selecting data, problem of storing the model) then using a model that directly incorporates regularisation (like LightGBM or XGBoost do) is the proper thing to do. Avoid direct feature selection. Particular for Gradient Boosting Machine methods I would also note that there is no single accepted way to define feature importance (e.g. e can look into how many times a feature is used for splitting or what is the feature's permutation importance) It would therefore be prudent to ensure that a common methodology is used.