My Question: Why does random forest consider random subsets of features for splitting at the node level within each tree rather than at the tree level?

Background: This is something of a history question. Tin Kam Ho published this paper on constructing "decision forests" by randomly selecting a subset of features to use for growing each tree in 1998. Several years later, in 2001, Leo Breiman published his seminal Random Forest paper, wherein the feature subset is randomly selected at each node within each tree, not at each tree. While Breiman cited Ho, he did not specifically explain the move from tree-level to node-level random feature selection.

I'm wondering what specifically motivated this development. It seems that selecting the feature subset at the tree level would still accomplish the desired decorrelation of the trees.

My theory: I haven't seen this articulated elsewhere, but it seems like the random subspace method would be less efficient in terms of getting estimates of feature importance. To obtain estimates of variable importance, for each tree, the features are randomly permuted one by one, and the increase in misclassification or increase in error for the out-of-bag observations is recorded. The variables for which the misclassification or error increase resulting from this random permutation is high are those with the greatest importance.

If we use the random subspace method, for each tree, we are only considering $m$ of the $p$ features. It may take several trees to consider all $p$ predictors even once. On the other hand, if we consider a different subset $m_i$ of the $p$ features at each node, we will consider each feature more times after fewer trees, giving us a more robust estimate of the feature importance.

What I've looked at thus far: So far, I have read Breiman's paper and Ho's paper, and done a broad online search for comparisons of the methods without finding a definitive answer. Note that a similar question was asked before. This question goes a bit further by including my speculation/work toward a possible solution. I would be interested in any answers, relevant citations, or simulation studies comparing the two approaches. If none are forthcoming, I plan to run my own simulation comparing the two methods.

  • 1
    $\begingroup$ I won't cite any references, so let's just call this a comment. If you are trying to understand what variables are useful, it may be the case that a specific variable is critical, but only on a small piece of the data. You could find this out with bagging the variables at the node level. You would never discover this with bagging at the tree level. $\endgroup$ – aginensky Jul 19 '18 at 17:01
  • 1
    $\begingroup$ I'm sure that Breiman has a comment related to this in his (imho) seminal paper, 'Statistics- The Two Cultures'. His point there is that sometimes the importance of a variable is masked by another variable. Bagging at the node level will allow one to see the what and when for a variable. $\endgroup$ – aginensky Jul 19 '18 at 17:03
  • $\begingroup$ Thanks for the comments. Getting back to my idea about efficiency: suppose a pair of variables were related and, as you said, the importance of one "masked" the importance of another. If we built an RF predictor with enough trees and used the tree-level feature subsetting, would we not eventually have enough trees with the "masked" feature and without the "masking" feature to get at the importance of the former without the impact of the latter? I do think we are talking about roughly the same idea at least. Thanks! $\endgroup$ – djlid Jul 19 '18 at 17:08
  • 2
    $\begingroup$ You might, but consider how many more trees you would have to build ! It's also not clear. Variable A might cause splits such that in none of them variable B will shine. It's just clearly intrinsically more robust to sample at the node level. To me, it relates to fundamentally what bootstrapping should be. $\endgroup$ – aginensky Jul 19 '18 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.