1
$\begingroup$

I am new to Machine Learning and would like to know is it always true that decision tree-based bagging has worse predictive performance and is slower running time than random forest?

$\endgroup$
1
$\begingroup$

Decision tree based bagging and a random forest are extremely similar approaches to modelling.

Most of the differences will be probably due to the fact that Random Forests routines will already be designed with the ability of tune some of the hyper-parameters that can associated with random sampling the original sample (e.g. proportions of the $p$ features to use) while when using standard bagging we would (almost always) assume that all of the $p$ the features available are to be included. Otherwise most of the other options (e.g. what would be the minimal data per leaf) would equally affect the (standard) decision trees that are used as base learners of both methods. Notice that some decision tree implementations by design do some random feature selection themselves (e.g. stopping after the first $\sqrt{p}$ features are examined given that one split is a valid based on other criteria), so in such cases the advantage of RF is partially nullified).

The above being said, usually random forest is better than bagging just because it can optimise the fraction of features used. If though the optimal fraction is large (e.g. 90% or less) potentially there are little to no gains to be had in terms of predictive performance. Similarly just because more features translates to more options to be explored, bagging will probably be a bit slower especially when the fractions of features used by the RF is low (e.g. 50% or less).

A final note: having "too many options to fiddle with" can be a problem at times. For example, usually two of the core features to regularise our ensemble fit are "max_tree_depth" and "min_leaf_data"; namely the maximum depth of our base learner and the minimum number of data-points on a leaf. Potentially, spending time on our RF optimising the fraction of features to use might distract us from optimising more relevant hyperparameters.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.