Feature importance understanding

Question

I would like to ask if I understand correctly the feature importance in random forest. I am examine random forest by selecting 4 or 6 features and also with different number of trees. I would like to know if I get a result like using 25, 50, 75, 100 trees with 4 features and 6 features. the result for having 25 tree with 4 features better because those randomly selected features were more important than when build a model with 75 trees? Is it correct or I completely misunderstand feature importance? many thanks

Answer 1

Feature importance or variable importance is a broad but very important concept in machine learning. Specifically, in terms of RF, your understanding is unfortunately problematic.

First, random forest is a parallel ensemble method, you grow trees parallelly using bootstrapped data. That means, having more trees in your forest doesn't necessarily associate to a worse performance, on the contrary, it would usually reduce overfitting. With that said, you might want to do a solid cross validation procedure in order to assure the performances.

Second, feature importance in random forest is usually calculated in two ways: impurity importance (mean decrease impurity) and permutation importance (mean decrease accuracy). The impurity importance of each variable is the sum of impurity decrease of all trees when it is selected to split a node. Permutation importance of a variable is the drop of test accuracy when its values are randomly permuted. It seems you interpret important features as having less trees but better performance (if not, you may need to clarify your question). As said before, larger number of trees in forest actually can be more beneficial.

Lastly, feature importance is algorithm and data dependent, so it is suggestive. You need to understand how it is computed to actually use it in practice.

Answer 2

I found this article to be one of the best explainations of feature importance with random forest.