1
$\begingroup$

I was reading this paper on the history of Bagging Estimators (https://www.stat.berkeley.edu/~breiman/bagging.pdf) and came across the following question (Why is bagging stable classifiers not a good idea?). Over here, the following point is mentioned:

enter image description here

Apparently, there are theoretical reasons that suggest that bagging "stable models" (e.g. Random Forests) is not advisable as this may in fact result in a loss in accuracy.

While this may be true, I noticed that the "randomForest" library in R (one of the most popular implementations of the Random Forest) has a function which allows you to directly combine several Random Forest models together (this looks like we are "combining" the three models together, not "bagging"):

# https://cran.r-project.org/web/packages/randomForest/randomForest.pdf
library(randomForest)
data(iris)
rf1 <- randomForest(Species ~ ., iris, ntree=50, norm.votes=FALSE)
rf2 <- randomForest(Species ~ ., iris, ntree=50, norm.votes=FALSE)
rf3 <- randomForest(Species ~ ., iris, ntree=50, norm.votes=FALSE)
rf.all <- combine(rf1, rf2, rf3)
print(rf.all)

I was just trying to better understand this point : If it is not advisable to bag together several "stable models" such as Random Forest - is there any reason that one of the main implementations would allow for these models to be "combined" together? Or do these theoretical warnings only refer to "bagging" stable models together and not "combining" stable models together?

Thanks!

$\endgroup$

1 Answer 1

4
$\begingroup$

The "combining" that is done in your code is just collecting the three random forests together into a single object, as a concatenation of the trees in each forest. The object rf.all is just a container to hold the trees: rf1 holds trees indexed as 1-50 (say), rf2 holds 51-100 and rf3 holds trees 101-150.

The combine method does not have any relationship to the quoted passage that is distinct from averaging the results of a single random forest object (e.g. the prediction of the ensemble rf1). In other words, if it's true that a single ensemble incurs a loss of accuracy, then it's true that combination of several ensembles must also share the same defect.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.