5
$\begingroup$

I am using scikit-learn to find feature importances using ExtraTreesClassifier and RandomForestClassifier, both of which have feature_importances_ attribute.

The data has 4 numeric predictors, 2 binary predictors and 2 multiclass (3 each) predictors which are converted to 6 dummy variables. There are about 12000 rows in dataset. The outcome variable is binary.

Following are the plots of features importances by 2 classifiers:

enter image description here

enter image description here

As can be seen by above figures, the importances are almost reverse of each other. SM,SL,SB and LM are most important by ET and almost least important by RF. Inverse is true for HT,WST,WT and A.

What could be the reason for such discrepancy? What does it tell about the data and the outcome variable? What is the best way to get feature importances reliably? Thanks for your insight.

Edit: I kept n_estimators=250. Results are similar with n_estimators=100

Improve this question
$\endgroup$
6
  • $\begingroup$ How many trees did you use? $\endgroup$
    – Sycorax ♦
    Commented Dec 8, 2018 at 1:49
  • $\begingroup$ I kept n_estimators=250. Results are similar with n_estimators=100. $\endgroup$
    – rnso
    Commented Dec 8, 2018 at 1:50
  • $\begingroup$ How is feature importance measured in the case of ET? There are different ways to measure feature importance in an RF generally, are RF and ET using the same methodology? (+1 fun question) $\endgroup$
    – usεr11852
    Commented Dec 16, 2018 at 23:48
  • $\begingroup$ As I mentioned in question above, I used feature_importance_ attribute: scikit-learn.org/stable/modules/generated/… I am not sure how that is calculated. $\endgroup$
    – rnso
    Commented Dec 17, 2018 at 2:07
  • 1
    $\begingroup$ Are you sure that your model is not overfitting? $\endgroup$
    – Jon Nordby
    Commented Jan 9, 2019 at 0:28

3 Answers 3

Reset to default
2
$\begingroup$

RandomForest and ExtraTrees will tend to supress redundant features. If a feature A is used already, and feature B does not add any new information over A it won't get used. And it will thus not contribute to feature importances, even though it is in general a highly predictive variable.

Which features gets picked as important among redundant features is random, and even the same model type is likely to give different feature importances in these cases each time it is trained. You can test this by passing different random_state variables.

Redundant features should have high mutual information. And they might be linearly correlated.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Which features here are likely to be linearly correlated? Are you suggesting I should check that and use only 1 amongst linearly correlated variables? $\endgroup$
    – rnso
    Commented Dec 17, 2018 at 2:11
  • 1
    $\begingroup$ "If a feature A is used already, and feature B does not add any new information over A it won't get used." This isn't true. Consider the case where A and B are identical copies. Random forests choose the variable to split from a random subset. A and B have equal probability of appearing in this subset, so sometimes A will be selected first, and sometimes B. The trees are fit independently. So, A and B would be expected to occur with equal frequency across trees in the forest, on average. Of course, there will be variability due to randomness from bagging and random feature selection. $\endgroup$
    – user20160
    Commented Dec 17, 2018 at 2:58
1
+50
$\begingroup$

Difference would be there since these two algorithms work in a different way.
ExtraTreeClassifier: fits a number of randomized decision trees (a.k.a. extra-trees) on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting.
whereas,
RandomForest: fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting. The sub-sample size is always the same as the original input sample size but the samples are drawn with replacement if bootstrap=True so please check if you put bootstrap=True or False.

Now the question is how can you decide what is the correct feature importance order, use three or four algorithms to find feature importance and let the majority wins! (almost every classification algorithm in scikit-learn offers feature importance)

Thanks!

Improve this answer
$\endgroup$
0
1
$\begingroup$
  1. We could take the model that has better accuracy on test set and consider its feature importance as better
  2. Compute feature importance from xgboost and compare the results
  3. Assuming that features SM/SL/SB and LM/LH/LG represent the 2 multi-classes, it looks like ET has grouped those variables together unlike RF
Improve this answer
$\endgroup$
1
  • $\begingroup$ Point 1 seems valid but is there any reference saying that better prediction is related to better feature importances, though this seems likely. Point 2: why only xgboost, why not many others also? Point 3 is just an observation. Points 2 and 3 are not really any answers. $\endgroup$
    – rnso
    Commented Dec 18, 2018 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.