3
$\begingroup$

One method to implement randomness in random forests is to use a random subset of the features for a split node.

What will be done at the next split node of the same branch beneath (that current / former split node)? Is it a random subset of the former subset, so, a subsubset? Because I think choosing a subset from the entire feature subset does not make sense?

$\endgroup$
2
  • 1
    $\begingroup$ "Because I think choosing a subset from the entire feature subset does not make sense?" Why not? $\endgroup$ Aug 7, 2021 at 14:25
  • $\begingroup$ Does the question mean this is the case? I would assume this would be much too random then? $\endgroup$
    – Ben
    Aug 8, 2021 at 14:00

2 Answers 2

2
$\begingroup$

At each split, you draw a new random sample of $m$ features.

From Hastie et al. Elements of Statistical Learning:

Algorithm 15.1 Random Forest for Regression or Classification.

  1. For $b = 1$ to $B$:

a. Draw a bootstrap sample $Z^*$ of size $N$ from the training data.

b. Grow a random-forest tree $T_b$ to the bootstrapped data, by recursively repeating the following steps for each terminal node of the tree, until the minimum node size $n_\min$ is reached.

i. Select $m$ variables at random from the $p$ variables.

ii. Pick the best variable/split-point among the $m$.

iii. Split the node into two daughter nodes.

  1. Output the ensemble of trees $\{T_b\}^B_1$.

To make a prediction at a new point $x$:

  • Regression: $\hat{f}^B_\text{rf}(x) = \frac{1}{B}\sum_{b=1}^B T_b(x).$

  • Classification: Let $\hat{C}_b(x)$ be the class prediction of the $b$th random-forest tree. Then $\hat{C}^B_\text{rf} (x) = > \textit{majority vote} \{\hat{C}_b(x)\}^B_ 1$.


Here's an example. You have a model with $p=4$ features, numbered 0, 1, 2, 3. You've set $m=2$, and at the first split you randomly draw 1,3. Then at the second split, you randomly draw 0,1. Then at the third split you randomly draw 1,3 again. In this example, choosing feature 1 in all three examples is purely a random coincidence. Likewise, so is choosing the same pair 1,3 twice also a random coincidence.

$\endgroup$
5
  • $\begingroup$ Thank you! The fact, that the drawn sample is a bootstrap sample means only that the samples are always drawn from the same set, or? $\endgroup$
    – Ben
    Aug 9, 2021 at 6:43
  • $\begingroup$ @Ben Bootstrapping is a statistical method for estimating the sampling distribution of an estimator by sampling with replacement from the original sample. More information: stats.stackexchange.com/questions/tagged/bootstrap $\endgroup$
    – Sycorax
    Aug 9, 2021 at 6:54
  • $\begingroup$ Ok, so I can understand this as a yes? :) Or is the RF really trying to estimate the distribution? I can't see it in the above quoted principle, or? $\endgroup$
    – Ben
    Aug 9, 2021 at 7:39
  • 1
    $\begingroup$ Yes, bootstrapping is done by resampling the same data; the distribution you're estimating with a random forest that uses bootstrapping is the distribution of random trees. It seems like you have a new, distinct question that is unrelated to understanding how to randomly select features. This is fine; you can ask a new question. $\endgroup$
    – Sycorax
    Aug 9, 2021 at 8:01
  • $\begingroup$ Thank you, will do that! $\endgroup$
    – Ben
    Aug 9, 2021 at 10:01
1
$\begingroup$

Looking at Chapter 15, second edition of Hastie, Tibshirani, Friedman "Elements of Statistical Learning", there is no subsubsetting. All features are available at all steps (by this mean that all features can be in the random feature subset that is used at each step - this is not constrained to those that were in the earlier one). This is well explained in the answer by Sycorax, but I leave this one here for the following:

As there are various schemes around, I can't guarantee that subsubsetting is nowhere done, however I don't see the point of it. I don't see the benefit of constraining randomness in this way. Particularly if some variables are really important and some others are not, it seems to be a waste to constrain later splits to stick to the variables already used before, which ultimately will mean that in the overall ensemble the variables are less uniformly covered, which is not good as we'd like to learn about all of them.

$\endgroup$
3
  • 2
    $\begingroup$ "there is no subsubsetting. All features are available at all steps" This is incorrect. Random forests consider a random subset of features on each split. This is described in the book chapter you mentioned (e.g. see section 15.2: 'Definition of random forests' and algorithm 15.1) $\endgroup$
    – user20160
    Aug 8, 2021 at 15:55
  • $\begingroup$ What I mean is that all features are available for the random subset. There's no subsubsetting in the sense of the original question. Sycorax explained this well, so I may as well delete my response. $\endgroup$ Aug 8, 2021 at 20:14
  • $\begingroup$ I've edited it; hopefully clearer now. $\endgroup$ Aug 8, 2021 at 20:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.