7
$\begingroup$

Each tree of a random forest is learned on a random bootstrapped sample. Consequently, given that the number of trees is large, it is probable that every observation of a data set is used to form at least one tree of a forest and hence there exists no "independent" sample which can be used for evaluating the performance of the random forest.

Which samples are used by random forest to calculate variable importance then? In my understanding there is no guarantee that there exists a sample which is independent from forming the trees.

$\endgroup$
2
  • $\begingroup$ There are several methods that can be used to calculate the variable importance. For example one can calculate "total decrease in node impurities from splitting on the variable, averaged over all trees", in which case the answer to you question would be "all the samples". Do you have a specific method in mind? $\endgroup$
    – psarka
    May 18, 2015 at 9:59
  • $\begingroup$ @Julian If you feel my post below answered your question, please mark it as 'accepted' by clicking the green check mark. If not, please add a comment to clarify. Thanks in advance. $\endgroup$
    – Antoine
    Aug 10, 2015 at 14:17

1 Answer 1

3
$\begingroup$

After each tree is grown, the values of a given predictor are randomly permuted in the out-of-bag sample (the one third of unique observations that are not part of the bootstrap sample) and the prediction error of the tree on the modified OOB sample is compared with the prediction error of the tree on the untouched OOB sample.

This process is repeated for all input variables, and averaged over all trees. Finally, variables are given scores proportional to the overall decrease in accuracy that their permutation induced.

The most important variables are the ones leading to the greatest losses in accuracy when “noised-up” (see Breiman 2001).

In short, you do have independent samples at the tree level. The importance of a given variable is first computed at the tree level, and the scores are aggregated over all trees to obtain the final, global importance score for the variable.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you for your answer, what is the advantage of randomly permuted variable importance over calculating the decrease in accuracy by completely removing that variable from the equation? E.g. let's say I have an ordinal variable called age_group. I can recalculate my model's accuracy by permuting the values of age_group across different observations, or seven simpler I can calculate the accuracy for the model by completely removing age_group as an independent variable. How and why is the former better? $\endgroup$
    – Zhubarb
    Aug 24, 2015 at 11:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.