I'm looking into using feature importance scores from GBDT for feature selection. Although GBDT does not need manual feature selection, the number of features is a restriction of the production system I'm dealing with. So I'm trying to reuse XGBoost for both feature selection and final prediction.
The problem is, XGBoost assigns high feature importance to uninformative variables. "Feature importance" here is the number of times a feature occurs in splits.
How to set hyperparameters such that XGBoost would avoid splits on uninformative features in training and, thus, produce distinctive importance scores that can be used in feature selection later?
For one case, I make a model with
sklearn.datasets.make_regression data. This data is generated by a function
y := b_0 * f0 + ... + b_9 * f9 with arbitrary coefficients
f2, f6, f9 are informative features with
b_i != 0 and other features having
b_i = 0. Features do not correlate with each other.
The model based on the original data returns proper importance scores. Well, after all
y := b_2 * f2 + b_6 * f6 + b_9 * f9 and it fits well.
Then I create a modified model. I add one standard deviation of noise to the feature
f2 in the original data and retrain the model.
The modified model returns quite different scores. Importance scores from
f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 original 5 0 309 1 1 0 105 2 1 267 modified 42 42 172 22 34 50 72 33 38 170
Problems with the modified model:
- Uninformative features get high scores (up to 70% of an informative feature)
- All informative features (
f2, f6, f9) lose scores, including features not affected by noise
Removing informative features changes the importance scores similarly to adding noise to informative features.
Adding noise to uninformative features does not change importance scores.
My understanding is that any change in informative features breaks the fully specified model
y(f2, f6, f9) and XGBoost tries to split on the other features.
Unlike XGBoost, OLS produce more robust results. OLS regression coefficients:
const f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 original -0.0 -0.0 -0.0 64.2 -0.0 -0.0 -0.0 5.3 -0.0 0.0 26.9 modified -0.6 0.0 -0.8 32.1 0.0 -0.1 -0.8 5.8 0.6 -0.1 27.6
2. Possible solutions
As for feature selection, the gain metric separates informative features better than occurrences do. Output of
f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 original 120.0 NaN 717941.0 527.0 51.0 NaN 12830.0 42.0 29.0 139537.0 modified 11058.0 12531.0 648796.0 10925.0 12703.0 11871.0 34828.0 12279.0 11783.0 247278.0
median(gains) * 2 would cut uninformative features off completely.
3. Similar questions
The questions that covered unstable feature importance:
- Why gradient boosting/random forest generate "unstable" feature importance?
- Unstable variable importance ranking
- Xgboost Feature Importance shift
- Feature importance in gradient boosted trees
- Unstable feature importance and optimized mtry values in Random Forest
This question differs: (1) features are uncorrelated; (2) the samples are identical, except one column; (3) uninformative features get large importance scores in the modified dataset.