1
$\begingroup$

I understand that we select a subset of predictors and data points to build each tree in the random forest. The tree is fully grown till the terminal nodes are pure. Can there be a chance that for a given set of predictors and data points, terminal nodes cannot be made pure, within a prespecified tree depth. Does random forest algorithm simply ignore such trees?

$\endgroup$
  • $\begingroup$ This is the typical case, if all your nodes are pure you have overfit the data. See this (slide When to stop splitting?): cse.msu.edu/~cse802/DecisionTrees.pdf $\endgroup$ – Bar Aug 18 '17 at 22:27
1
$\begingroup$

Trees in random forests are very deep, and indeed typically grown until the terminal nodes are pure. A lot of these splits are overfit. The overfitting averages out when the predictions are averaged.

You can illustrate this by growing a random forest with only random noise as predictors. The model will be forced to use the noise for splitting.

By the way, a new subset of predictors is randomly selected for each split, not for each tree.

$\endgroup$
  • $\begingroup$ Thanks for making it clear. My intended question was we usually prespecfiy the maximum tree depth in R code. What happens if pure nodes are not achieved within the tree depth? $\endgroup$ – kasa Aug 19 '17 at 4:42
  • $\begingroup$ Doesn't matter; in the case of a classification problem, the impure node will predict the most common value in that node for that combination of predictors, and in the case of regression, it will predict the mean if the node. I imagine that in case of a tie in a classification forest, one of the tied categories is returned at random... It shouldn't matter much. That's just one tiny node in a forest. $\endgroup$ – Caspar van Lissa Aug 19 '17 at 7:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.