1
$\begingroup$

in Elements of Statistical Learning chapter 15. Random Forest, we see authors' note on RF v.s. GBT. One of them is that at 1000 terms, GBM depth 4 has smaller error than RF depth 6. Also we notice RF depth 6 has smaller error than RF depth 2.

This led me to think that does random forest trees need to be set deeper than trees in GBT? If so, what would be the mathematical proof to it?

One guess I have is in each RF tree we only use a subset of variables, hence to achieve same descent in error naturally we need more depth, but not sure if my direction is correct. Thanks for your inputs!

$\endgroup$
2
$\begingroup$

This is strongly connected with bias variance trade off and with the greedy nature of decision trees. A decision tree with large depth will tend to have large variance because it happens often that another tree to be very different, but might have small bias because it is very local (small regions). In contrast a tree with small depth does not vary that much but can have large bias because it might be not so complex. Now, random forests uses bagging, which is model averaging. Averaging reduces mostly the variance. So rf are good to reduce deep trees, it is not so effective on small one. Boosting uses gradients, which means going in small steps to target. If the tree is deep, it might go in a local minima very soon, so it’s better to have a much global view. This is doable better with shallow trees because of stability and myopic view, which is equivalent with a global viw

$\endgroup$
  • $\begingroup$ +1. But averaging reduces only variance! $\endgroup$ – bi_scholar Jun 24 at 17:44
  • $\begingroup$ Hi rapaio! thanks for the answer. I think your point of having a global view makes a lot of sense, but I'm still not sure why would rf need a deeper tree to have similar error as GBT mathematically $\endgroup$ – numerairX Jun 24 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.