3
$\begingroup$

Does anyone know what the actual calculation behind the feature importance (importance type='gain') method in the xgboost library is? I looked through the documentation and also consulted some other pages but I couldn't find an exact reference on what the actual calculation behind the measures is.

I would be glad for any kind of scientific references of the calculation method as I'd like to cite it.

$\endgroup$

1 Answer 1

4
$\begingroup$

Like with random forests, there are different ways to compute the feature importance. In XGBoost, which is a particular package that implements gradient boosted trees, they offer the following ways for computing feature importance:

How the importance is calculated: either “weight”, “gain”, or “cover”
- ”weight” is the number of times a feature appears in a tree
- ”gain” is the average gain of splits which use the feature
- ”cover” is the average coverage of splits which use the feature where coverage is defined as the number of samples affected by the split

(Source: https://xgboost.readthedocs.io/en/latest/python/python_api.html)

Now, the gain is basically just the information gain averaged over all trees. For that, given a node in the tree, you first compute the node impurity of the parent node -- e.g., using Gini or entropy as a criterion. Then, you compute the node impurities of the child nodes if you were to use a given feature for the split. Finally, the information gain is calculated by subtracting the child impurities from the parent node impurity. Let me know if you need more details on that.

$\endgroup$
3
  • $\begingroup$ Thank you for your response. Based on your answer, my follow-up question would then be if the feature importance of xgboost is truly identical with the calculation of feature importance in random forests or are there any differences? Like I said, I'd like to cite something on this topic but I cannot cite any SO answers or Medium blog posts whatsoever. Therefore, some paper or any official writing what calculation is used would be helpful. This is not to say that I don't believe you :) $\endgroup$
    – TheDude
    Aug 16, 2019 at 13:32
  • 1
    $\begingroup$ @TheDude Even if the computations are the same, xgboost is a different model from random forest so the feature importance metrics won't be identical in general. $\endgroup$
    – Sycorax
    Aug 16, 2019 at 13:45
  • 2
    $\begingroup$ Since XGBoost is a particular software implementation of gradient boosting, the only official resources you might find are the original paper (delivery.acm.org/10.1145/2940000/2939785/p785-chen.pdf) and the documentation I linked above. I think citing the paper for how Gain is computed and citing the API docs for how the feature importance is computed based on the Gain should be sufficient?! $\endgroup$
    – resnet
    Aug 16, 2019 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.