0
$\begingroup$

Per my understanding, there are 2 kinds of "subsets" that can be used when creating trees: 1) Subset of the dataset, 2) Subset of the features used per split.

The concepts that I'm comparing are: 1) Bagging, 2) Random Forest, and 3) Boosting.

Please let me know if the following is correct or incorrect:

Bagging: Uses Subset of the dataset (bootstrapping) to create trees, but All features can be used for splits.

Random Forest: Uses Subset of the dataset (bootstrapping) to create trees, and only Subsets of features can be used for splits.

Boosting: Uses Entire dataset to create trees (no bootstrapping), and All features can be used for splits.

Yes I know there are more differences between the three, but wanted to hone in on these subset differences here and clarify my understanding on these first.

$\endgroup$
2
  • 1
    $\begingroup$ Sounds like a good summary. Maybe it is worth noting that boosting is also possible with both types of subsetting. $\endgroup$
    – Michael M
    Jan 19, 2023 at 21:50
  • $\begingroup$ Breiman's clever coining of words in 2001 is the original, go-to work on these methodologies. stat.berkeley.edu/~breiman/randomforest2001.pdf $\endgroup$
    – user78229
    Mar 12 at 0:06

1 Answer 1

0
$\begingroup$

Random forests use bagging (bagging is just a contraction of "bootstrap aggregation," see Elements of Statistical Learning Section 8.7). Bagging draws a bootstrap sample of the data (randomly select a new sample with replacement from the existing data), and the results of these random samples are aggregated (because the trees' predictions are averaged). But bagging, and column subsampling can be applied more broadly than just random forest. (There's also a discussing in ESL of how random forest is well-positioned to benefit from bagging, while other learning methods are not.)

The boosting implementations that I'm familiar (e.g. xgboost) will also support random subsampling of columns. But your guess is correct, in the sense that column subsampling is really incidental to the boosting procedure itself. Boosting is about estimator $T+1$ "correcting" the errors of the previous $T$ estimators. Likewise, bagging is also incidental to the boosting procedure, but bootstrap resampling of the data could be implemented as well.

$\endgroup$
3
  • $\begingroup$ So sounds like the bagging, random forest, and boosting above are correct, and you would add: XGBoost/GBT: Uses Entire dataset to create trees (no bootstrapping), and either All or Subset of Features can be used for splits. $\endgroup$
    – Katsu
    Jan 19, 2023 at 21:07
  • $\begingroup$ You can use resampling with boosting models. For example, the xgboost documentation xgboost.readthedocs.io/en/stable/… lists several options for sampling training observations. $\endgroup$
    – Sycorax
    Jan 19, 2023 at 21:10
  • $\begingroup$ Gotcha, edited to: XGBoost/GBT: Uses Entire or Subset of Dataset to create trees (bootstrapping optional), and All Features can be used for splits. $\endgroup$
    – Katsu
    Jan 19, 2023 at 21:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.