7
$\begingroup$

I'm trying to use a build in function in XGBoost to print the importance of features. My code is like

import xgboost as xgb
...
clf_xgboost = xgb.XGBClassifier(...)
clf_xgboost.fit(X, y)
for type in ['weight', 'gain', 'cover']:
  print(clf_xgboost.get_booster().get_score(importance_type=type))

The program prints 3 sets of importance values. Each set looks like, {'feature1':0.11, 'feature2':0.12, ...}

And I googled the importance_type and found this page. The page gives a brief explanation of the meaning of the importance types.

Besides the page also say clf_xgboost has a .get_fscore() that can print the "importance value of features".

Can someone explain the difference between .get_fscore() and .get_score(importance_type)? And the difference between the 3 importance_types?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
7
$\begingroup$

In XGBoost library, feature importances are defined only for the tree booster, gbtree. So, I'm assuming the weak learners are decision trees.

get_fscore uses get_score with importance_type equal to weight.

The three importance types are explained in the doc as you say. I could elaborate on them as follows:

weight: XGBoost contains several decision trees. In each of them, you'll use some set of features to classify the bootstrap sample. This type basically counts how many times your feature is used in your trees for splitting purposes.

gain: In R-Library docs, it's said the gain in accuracy. This isn't well explained in Python docs. I think, this option could be easily confused with Information Gain used in decision tree node splits.

cover: In each node split, a feature splits the dataset falling into that node, which is a proportion of your training observations. So, your selected feature concerns some portion of the dataset.

Improve this answer
$\endgroup$
6
  • 2
    $\begingroup$ Don't trust any of these importance scores unless you bootstrap them and show that they are stable. For this you'd need to bootstrap the entire process, i.e. any steps that used supervised learning. You will often be surprised that importance measures are not trustworthy. $\endgroup$
    – Frank Harrell
    Commented Mar 12, 2019 at 11:36
  • 3
    $\begingroup$ Agree. My answer aims only demystifying the methods and the parameters associated, without questioning the value proposed by them. $\endgroup$
    – gunes
    Commented Mar 12, 2019 at 11:39
  • 1
    $\begingroup$ @FrankHarrell can you elaborate on your comment a little more? I'd love to hear what can be done to strengthen my trust in the feature importance results I see. $\endgroup$
    – Mariah
    Commented Aug 16, 2021 at 10:07
  • $\begingroup$ You can't do much about lack of information. Confidence limits for variable importances expose the difficulty of the task and help to understand why selecting variables (dropping variable) using supervised learning is often a bad idea. $\endgroup$
    – Frank Harrell
    Commented Aug 16, 2021 at 12:31
  • 1
    $\begingroup$ Take those B scores and compute their 0.025 and 0.975 quantiles, which will give you the bootstrap nonparametric percentile confidence limits for variable importance. Alternatively rank variables by importance at each bootstrap and compute bootstrap confidence interval for each variable's rank. Examples in high dimensional data analysis chapter of BBR. $\endgroup$
    – Frank Harrell
    Commented Aug 17, 2021 at 12:16
0
$\begingroup$

Note they also expose vaguely defined feature_importances_:

Feature importances property, return depends on importance_type parameter. When model trained with multi-class/multi-label/multi-target dataset, the feature importance is “averaged” over all targets. The “average” is defined based on the importance type. For instance, if the importance type is “total_gain”, then the score is sum of loss change for each split from all trees

I don't understand what the output should be for binary classification, it's not "multi" in common sense, but it differs from all the get_score options.

Also I found that training XGBoost with enable_categorical=True is a total disaster (at least version 1.7.6). Categorical factors totally unrelated to the target get the highest feature importance among all the variables, including really impactful ones. Catboost and LightGBM handle categoricals much more correctly.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.