1
$\begingroup$

Is there a difference between using a bagging classifier with base_estimaton=DecisionTreeClassifier and using just the RandomForestClassifier? This question refers to models from python library called sklearn.

$\endgroup$
0

1 Answer 1

5
$\begingroup$

Yes, there is a difference. In sklearn if you bag decision trees, you still end up using all features with each decision tree. In random forests however, you use a subset of features.

The official sklearn documentation on ensembling methods could have been a bit more clear about the difference, here is what it says:

  1. "When samples are drawn with replacement, then the method is known as Bagging"
  2. "In random forests (see RandomForestClassifier and RandomForestRegressor classes), each tree in the ensemble is built from a sample drawn with replacement (i.e., a bootstrap sample) from the training set."

So it would appear there is no difference if you bag decision trees, right? It turns out, the documentation also states:

"Furthermore, when splitting each node during the construction of a tree, the best split is found either from all input features or a random subset of size max_features."

So this is one more way of introducing randomness, by limiting the number of features at the splits. In practice, it is useful to indeed tune max_features to get a good fit.

$\endgroup$
2
  • 2
    $\begingroup$ It's not each tree that receives a subset of candidate features, it's each split. $\endgroup$ Commented Apr 18, 2020 at 14:31
  • $\begingroup$ @Matthew Drury Thank you for point out, corrected. $\endgroup$
    – PAF
    Commented Apr 18, 2020 at 16:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.