I know that using Bagging method in a RF, implies that the subset we give to the root node of each tree, has randomly selected Features and Attributes.

I also know that during the split of a node in a RF we split only by feature selection, but does Bagging has to do anything with the process of splitting a node.

Or does its responsibility end in the Feature, Attribute selection for the subset of a single tree?


The short answer is no, bagging - as a process - is not directly tied to Random Forest's way of splitting a node. The purpose of bagging is to reduce overall model variance, whereas Random Forest's additional random feature selection is meant to produce uncorrelated predictions.

Bootstrap aggregation (bagging) can be described as random subsampling with replacement. Every subsample is used to train a weak learner and, in a separate step, all weak learners are aggregated to produce a strong learner; i.e. one that produces predictions with reduced variance. Bagging is a generic concept in ensemble learning and does not specifically involve Random Forest's feature selection. Without random feature selection, instead of Random Forest you will get Bagged Decision Trees.

Strictly speaking, bagging is not really tied to decision trees at all. Bagging generally works better with an ensemble of weak learners with high variance (prone to overfitting), and simple CART models happen to fall under that category. However, in theory, any weak learner that uses greedy stochastic search or simply produces high variance predictions, should be a good candidate for bagging.

  • 1
    $\begingroup$ "bagging has nothing to do with Random Forest's process of splitting a node". This isn't true. Random forests use bagging to train the trees. Because the node splits are chosen to approximately optimize performance on resampled/bagged data, bagging does affect the choice of splits. Indeed, if this weren't the case, there would be little reason to do it. It's also worth pointing out that subsampling features (as RFs do when splitting nodes) is sometimes called "feature bagging", although this is distinct from resampling data points. $\endgroup$
    – user20160
    Feb 8 '19 at 22:53
  • $\begingroup$ Sorry @user20160, I don't see your point at all. Bagging affects the choice of splits in the sense that it samples data used to train trees. Bagging does not train individual trees, CART does. Feature bagging is again a completely different thing that uses bagging for a different reason (my point from the start). The reason to do each of those things is already explained in my answer. $\endgroup$
    – Digio
    Feb 8 '19 at 23:02
  • $\begingroup$ I don't mean that each individual tree is itself bagged. Rather, that it's trained on data that has been resampled with replacement, which is part of the bagging process used to define the entire ensemble. The split points chosen by CART will be different on this resampled data than they would have had resampling not been used. So, the overall bagging process used by RFs affects the splits of the individual trees (via the resampling step). My objection here is just with the first sentence. $\endgroup$
    – user20160
    Feb 8 '19 at 23:38
  • $\begingroup$ I just rephrased it for the sake of clarity, but also, please feel free to edit. $\endgroup$
    – Digio
    Feb 8 '19 at 23:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy