I'm working on a text classification problem and I am comparing LightGBM and XGBoost performances. Both on train and test sets I get roughly the same accuracy metrics, but what looks amusing to me is that feature importances (as well as shapley values and permutation importances) look quite different (some of the most important features for one algorithm are never used by the other and viceversa).
Of course there may be some collinearity between my variables, but I don't get how one may choose one variable over another.
And here's my question: is there any way of soundly combining the two? I don't have that many data to stack them btw. By combining I mean anything from feature selection to prediction combining.