" The fundamental difference between bagging and random forest is that in Random forests, only a subset of features are selected at random out of the total and the best split feature from the subset is used to split each node in a tree, unlike in bagging where all features are considered for splitting a node." Does that means that bagging is the same as random forest, if only one explanatory variable (predictor) is used as input?
$\begingroup$
$\endgroup$
2
asked Feb 26, 2017 at 13:44
-
$\begingroup$ Theoretically yes, but that kind of case (n_features = 1) does not make a lot of sense, right? $\endgroup$– RockbarCommented Feb 10, 2023 at 13:56
-
$\begingroup$ Does that mean bagging will always outperform against random forest but takes more time? @Rockbar $\endgroup$– Sazzad Hissain KhanCommented Oct 19, 2023 at 10:56
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
1
The fundamental difference is that in Random forests, only a subset of features are selected at random out of the total and the best split feature from the subset is used to split each node in a tree, unlike in bagging where all features are considered for splitting a node.
-
$\begingroup$ So if we have bagging models with logistic reg, linear reg, three decision tree as base models all three decision tree will use all features? $\endgroup$ Commented Jan 16, 2021 at 6:12
$\begingroup$
$\endgroup$
2
I would like to provide clarification, there is a disctinction between bagging and bagged trees.
Bagging (bootstrap + aggregating) is using an ensemble of models where:
- each model uses a bootstrapped data set (bootstrap part of bagging)
- models' predictions are aggregated (aggregation part of bagging)
This means that in bagging, you can use any model of your choice, not only trees.
Further, bagged trees are bagged ensembles where each model is a tree.
So, in a sense, each bagged tree is a bagged ensemble, but not each bagged ensemble is a bagged tree.
Given this clarification, I think that user3303020's answer provides a good explanation.
-
$\begingroup$ There is a more important difference that you didn't mention. Random forest is not just bagged trees, it has randomness in feature selection of each tree. $\endgroup$– HadijCommented Apr 7, 2021 at 22:56
-
$\begingroup$ Nice. I think this is an important difference most people don't mention it. In Bagging, you have the option to select the algorithm (estimator), but in random first, it's always a decision tree. $\endgroup$ Commented Jun 29 at 17:57
$\begingroup$
$\endgroup$
Bagging in general is an acronym like work that is a portmanteau of Bootstrap and aggregation. In general if you take a bunch of bootstrapped samples of your original dataset, fit models $M_1, M_2, \dots, M_b$ and then average all $b$ model predictions this is bootstrap aggregation i.e. Bagging. This is done as a step within the Random forest model algorithm. Random forest creates bootstrap samples and across observations and for each fitted decision tree a random subsample of the covariates/features/columns are used in the fitting process. The selection of each covariate is done with uniform probability in the original bootstrap paper. So if you had 100 covariates you would select a subset of these features each have selection probability 0.01. If you only had 1 covariate/feature you would select that feature with probability 1. How many of the covariates/features you sample out of all covariates in the data set is a tuning parameter of the algorithm. Thus this algorithm will not generally perform well in high-dimensional data.
answered Feb 26, 2017 at 15:46