0
$\begingroup$

My understanding from the sklearn random forest classifier documentation is that at every node, a new sample of m features are sampled from a total of M. Some criterion is computed for each of them and the best one is selected to split on. My question is, are these features sampled with replacement?

Here they say they do Random forest tree growing algorithm and to stop at some number of samples in a leaf to be able to detect some relationships, but the default value for the min samples leaf parameter in the sklearn implementation is 1. If this is the case why would you split on the same feature twice?

My guess would be that they are not, such that at every node a new feature sample of size m is taken from M removed with all features previously split on, hence every tree wil either grow to depth of max M or stop once every leaf node is pure.

Which one of the two is it?

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
2
$\begingroup$

My question is, are these features sampled with replacement?

At each node, features are sampled without replacement. Sampling with replacement would effectively reduce the number of features sampled at each split, because the best split among some feature is the same for that feature sampled a second time.

Across the entire tree, you might choose the same feature more than once. This is because information about previous splits in no way informs how subsequent splits are chosen. Splitting multiple times on the same feature can be necessary to model relationships which are not step functions with a single step.

why would you split on the same feature twice?

Suppose you have a binary problem and class 1 only occurs when the value of a continuously-valued feature is in the middle of a range. You'd need 2 binary splits to isolate the middle range of values.

Improve this answer
$\endgroup$
4
  • $\begingroup$ So unless specified otherwise every tree will grow until all leaf nodes are pure or not separable? $\endgroup$
    – Lara
    Mar 4, 2020 at 21:58
  • $\begingroup$ @Lars Until purity or until there's no valid split, where valid split includes "not separable" as a special case. Hyperparameters govern other cases where the procedure may terminate, such as too few samples in a leaf, too few child samples, improvement is too small, etc. $\endgroup$
    – Sycorax ♦
    Mar 4, 2020 at 22:00
  • $\begingroup$ @SycoraxsaysReinstateMonica could you elaborate how not separable is a special case of no valid split existing? $\endgroup$
    – Lara
    Mar 4, 2020 at 22:38
  • $\begingroup$ I guess it just depends on how you want to group these things together; I suppose "purity" is another case of "no valid split" because it's impossible to divide a pure node into a node with lower impurity. The difference doesn't seem important, but my point is just that we need to include all termination criteria, including the ones that depend on hyperparameters. $\endgroup$
    – Sycorax ♦
    Mar 4, 2020 at 22:43

Not the answer you're looking for? Browse other questions tagged or ask your own question.