I'm looking into using feature importance scores from GBDT for feature selection. Although GBDT does not need manual feature selection, the number of features is a restriction of the production system I'm dealing with. So I'm trying to reuse XGBoost for both feature selection and final prediction.

The problem is, XGBoost assigns high feature importance to uninformative variables. "Feature importance" here is the number of times a feature occurs in splits.

How to set hyperparameters such that XGBoost would avoid splits on uninformative features in training and, thus, produce distinctive importance scores that can be used in feature selection later?

1. Example

For one case, I make a model with sklearn.datasets.make_regression data. This data is generated by a function y := b_0 * f0 + ... + b_9 * f9 with arbitrary coefficients b_i, where f2, f6, f9 are informative features with b_i != 0 and other features having b_i = 0. Features do not correlate with each other.

GBDT (XGBoost)

The model based on the original data returns proper importance scores. Well, after all y := b_2 * f2 + b_6 * f6 + b_9 * f9 and it fits well.

Then I create a modified model. I add one standard deviation of noise to the feature f2 in the original data and retrain the model.

The modified model returns quite different scores. Importance scores from get_score(importance_type='weight'):

          f0  f1   f2  f3  f4  f5  f6  f7  f8   f9
original   5   0  309   1   1   0 105   2   1  267
modified  42  42  172  22  34  50  72  33  38  170

Problems with the modified model:

  • Uninformative features get high scores (up to 70% of an informative feature)
  • All informative features (f2, f6, f9) lose scores, including features not affected by noise

Removing informative features changes the importance scores similarly to adding noise to informative features.

Adding noise to uninformative features does not change importance scores.

My understanding is that any change in informative features breaks the fully specified model y(f2, f6, f9) and XGBoost tries to split on the other features.


Unlike XGBoost, OLS produce more robust results. OLS regression coefficients:

          const   f0   f1    f2   f3   f4   f5   f6   f7   f8    f9
original   -0.0 -0.0 -0.0  64.2 -0.0 -0.0 -0.0  5.3 -0.0  0.0  26.9
modified   -0.6  0.0 -0.8  32.1  0.0 -0.1 -0.8  5.8  0.6 -0.1  27.6

2. Possible solutions

As for feature selection, the gain metric separates informative features better than occurrences do. Output of get_score(importance_type='gain'):

               f0       f1        f2       f3       f4       f5       f6       f7       f8        f9
original    120.0      NaN  717941.0    527.0     51.0      NaN  12830.0     42.0     29.0  139537.0
modified  11058.0  12531.0  648796.0  10925.0  12703.0  11871.0  34828.0  12279.0  11783.0  247278.0

A threshold median(gains) * 2 would cut uninformative features off completely.

3. Similar questions

The questions that covered unstable feature importance:

This question differs: (1) features are uncorrelated; (2) the samples are identical, except one column; (3) uninformative features get large importance scores in the modified dataset.

  • 1
    $\begingroup$ I might saying the obvious here but a GBDT is not a modelling panacea. Yes, it will make automatically feature selection up to a point but ultimately it is a predictive routine not a variable importance mechanism. For all we know the noise features help fight over-fitting. (Think of it as brutal ridge regression approach.) Unless we bootstrap the results to get some sense of their variance we cannot infer much here. Finally note that while you say "uninformative features get large importance scores in the modified dataset" the ranking of the features is still similar/correct. $\endgroup$ – usεr11852 says Reinstate Monic Dec 9 '18 at 0:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.