1
$\begingroup$

So it sounds like boosting techniques (eg. XGBoost) uses weak learners (stumps) to gradually learn sequentially. This is not in dispute I hope.

However, with bagging techniques (eg. Random Forest) I'm not sure if it grows weak learners (like stumps) in parallel, or fully grown trees in parallel? Since the definition of "Ensemble modeling" is using many weak learners to form 1 powerful model, I'm leaning towards Random Forests using stumps, but I can be swayed the other direction as well?

$\endgroup$
0

2 Answers 2

1
$\begingroup$

Xgboost can use deep or shallow trees (you can set a maximum tree depth). The default xgboost tree depth is 6; see the xgboost documentation.

Likewise, random forest can use deep or shallow trees for the same reason. For example, the implementation in sklearn.ensemble.RandomForestClassifier has no maximum depth by default, so the behavior is "nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples."

$\endgroup$
1
  • $\begingroup$ Perf thank you! $\endgroup$
    – Katsu
    Jan 20, 2023 at 19:18
2
$\begingroup$

The original motivation for random forests is to make decision trees less overfit, and in that sense fully grown trees are the expectation. However, in practice I find that some pruning still provides a benefit.

Stumps are quite unlikely to be optimal though. Unlike boosting, where each subsequent tree improves upon the current fit, bagging weak learners doesn't let them learn from each other. Most likely you'll end up with lots of the stumps being exactly the same: their bags just aren't different enough from each other to choose different splits.

$\endgroup$
3
  • $\begingroup$ Good clarification on stumps being suboptimal. You prune your random forests? Thought pruning only applies if ur only using single decision trees? $\endgroup$
    – Katsu
    Jan 22, 2023 at 1:04
  • 1
    $\begingroup$ @Katsu Anything that prevents a full tree can be called pruning. I had in mind "pre-pruning" methods: setting a maximum depth or number of leaves, or minimum number of samples per leaf or impurity improvement per split. $\endgroup$ Jan 22, 2023 at 2:02
  • $\begingroup$ Gotcha prepruning def makes sense! $\endgroup$
    – Katsu
    Jan 22, 2023 at 17:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.