Ok so I think I have listened to a few wrong discussions on random forests because now I have a very confused question. With respect to Random Forests and bagging/bootstrapping, I'm good there. The confusing part is how the random selection of attributes is sometimes called "boosting". Basically from what I know of boosting, it is making many weak learners strong learners, so there would be only a few attributes in a boosting tree and it would be combined with many other like trees to have a strong learner. So is this applied to random forests too? And if it is not, then how can the attribute selection of random forrests be said to be different than boosting.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
They are both ensemble methods. But rf treats each tree equally. You can train multiple trees in parallel. And the final result is just the average. But in boosting trees, trees are trained sequentially, that is new tree is trained on the residuals left by former predictor. In general, boosting has better performance but is easier to overfit and takes more time to train.
-
$\begingroup$ Thank You. So then can you characterize the random feature selection of random forests and how it is different than boosting? It is this aspect that I have heard called boosting before. When there are only a subset of attributes available for a split, can you exlain how this aspect is different than boosting? $\endgroup$ Commented Apr 18, 2016 at 5:19
-
$\begingroup$ Boosting has nothing to do with randomly sampling predictors. Boosting refers to making a sequence of weak predictions better by successively training on the residuals. You are boosting a very weak predictor into a strong ensemble predictor. $\endgroup$– Zelazny7Commented Apr 21, 2016 at 17:20
-
$\begingroup$ Forgot to add, that in Gradient Boosting, the feature selection is NOT random (although some implementations allow for sampling of features). At each stage, the best possible weak learner is created from all available features. Contrast this with RF where the features are randomly sampled at every node of the tree. $\endgroup$– Zelazny7Commented Apr 21, 2016 at 17:22