0
$\begingroup$

I would like to ask if I understand correctly the feature importance in random forest. I am examine random forest by selecting 4 or 6 features and also with different number of trees. I would like to know if I get a result like using 25, 50, 75, 100 trees with 4 features and 6 features. the result for having 25 tree with 4 features better because those randomly selected features were more important than when build a model with 75 trees? Is it correct or I completely misunderstand feature importance? many thanks

$\endgroup$

2 Answers 2

0
$\begingroup$

Feature importance or variable importance is a broad but very important concept in machine learning. Specifically, in terms of RF, your understanding is unfortunately problematic.

First, random forest is a parallel ensemble method, you grow trees parallelly using bootstrapped data. That means, having more trees in your forest doesn't necessarily associate to a worse performance, on the contrary, it would usually reduce overfitting. With that said, you might want to do a solid cross validation procedure in order to assure the performances.

Second, feature importance in random forest is usually calculated in two ways: impurity importance (mean decrease impurity) and permutation importance (mean decrease accuracy). The impurity importance of each variable is the sum of impurity decrease of all trees when it is selected to split a node. Permutation importance of a variable is the drop of test accuracy when its values are randomly permuted. It seems you interpret important features as having less trees but better performance (if not, you may need to clarify your question). As said before, larger number of trees in forest actually can be more beneficial.

Lastly, feature importance is algorithm and data dependent, so it is suggestive. You need to understand how it is computed to actually use it in practice.

$\endgroup$
2
  • $\begingroup$ Thank you so If i have to analyse my result for feature importance how I should write it. I did chose 4 feature and 6 feature with different number of trees and now I had a result of test accuracy. So how I thought it is wrong so I can't write that down. Books doesn't really tell about it. I did look at that impurity importance. but then with random forest if it is feature selection and have some feature more important and selecting randomly I thought it might affect the accuracy but then it does not. $\endgroup$
    – Zoe
    Aug 28, 2020 at 22:29
  • $\begingroup$ Try permutation importance, which is straightforward to understand and implement. Randomly selecting subsets of features is a way to do feature selection, that's fine, but you have to do the selection in an inner cross validation. Search for nested cross validation, you can find many good posts. $\endgroup$
    – doubllle
    Aug 29, 2020 at 10:34
0
$\begingroup$

I found this article to be one of the best explainations of feature importance with random forest.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.