I am using both random forest and xgboost to examine the feature importance. but i noticed that they give different weights for features as shown in both figures below, for example HFmean-Wav had the most important in RF while it has been given less weight in XGBoost and i can understand why?
First you should understand that these two are similar models not same ( Random forest uses bagging ensemble model while XGBoost uses boosting ensemble model), so it may differ sometimes in results. Now let me tell you why this happens.
When the correlation between the variables are high, XGBoost will pick one feature and may use it while breaking down the tree further(if required) and it will ignore some/all the other remaining correlated features(because we will not be able to learn different aspects of the model by using these correlated feature because it is already highly correlated with the chosen feature).
But in random forest , the tree is not built from specific features, rather there is random selection of features (by using row sampling and column sampling), and then the model in whole learn different correlations of different features. So you can see the procedure of two methods are different so you can expect them to behave little differently.
Hope this helps!
Also, in XGBoost the default measure of feature importance is average gain whereas it's total gain in sklearn. See, https://xgboost.readthedocs.io/en/latest/python/python_api.html#module-xgboost.sklearn
importance_type (string, default "gain") – The feature importance type for the feature_importances_ property: either “gain”, “weight”, “cover”, “total_gain” or “total_cover”.
and https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html
The importance of a feature is computed as the (normalized) total reduction of the criterion brought by that feature.
Specifying importance_type='total_gain' in XGBoost seems to produce more comparable rankings.
