why does random forest trees need to be deeper than gradient boosting trees

Question

in Elements of Statistical Learning chapter 15. Random Forest, we see authors' note on RF v.s. GBT. One of them is that at 1000 terms, GBM depth 4 has smaller error than RF depth 6. Also we notice RF depth 6 has smaller error than RF depth 2.

This led me to think that does random forest trees need to be set deeper than trees in GBT? If so, what would be the mathematical proof to it?

One guess I have is in each RF tree we only use a subset of variables, hence to achieve same descent in error naturally we need more depth, but not sure if my direction is correct. Thanks for your inputs!

Answer 1

This is strongly connected with bias variance trade off and with the greedy nature of decision trees. A decision tree with large depth will tend to have large variance because it happens often that another tree to be very different, but might have small bias because it is very local (small regions). In contrast a tree with small depth does not vary that much but can have large bias because it might be not so complex. Now, random forests uses bagging, which is model averaging. Averaging reduces mostly the variance. So rf are good to reduce deep trees, it is not so effective on small one. Boosting uses gradients, which means going in small steps to target. If the tree is deep, it might go in a local minima very soon, so it’s better to have a much global view. This is doable better with shallow trees because of stability and myopic view, which is equivalent with a global viw