I'm studying Random Forests, but after reviewing the methods I got the following line of reasoning: I feel like the big advantage of random forests happens in the bagging process where nearly uncorrelated predictions are created due to the random features, producing predictions with low variance. On the other hand, this method doesn't require your classification method to be a decision tree.

If other classification algorithms (LDA, QDA, logistic regression, etc) usually perform better than decision trees, why would I choose to use decision trees as my classifier?

Random forests seem a lot more popular than bagging with other classifiers (it even got its own name) but I don't see any particular reason why.

  • $\begingroup$ One reason might be ease of interpretation to non-ML savvy outsiders. $\endgroup$ – gannawag Sep 5 '18 at 1:26
  • $\begingroup$ Doesn't the bagging process kills the interpretation? Your 'final' prediction is an average of predictions produced by possibly (and very likely) different trees. $\endgroup$ – Freguglia Sep 5 '18 at 1:30
  • $\begingroup$ Yeah good point...I was thinking that decision trees are pretty easy to understand, and it might be easier to jump from there to bagging than starting with a more complicated approach. $\endgroup$ – gannawag Sep 5 '18 at 1:35
  • $\begingroup$ Ask yourself: what happens if you apply bagging to ordinary least squares? If you apply the parametric bootstrap, you will see that the center of the bootstrap distribution is just the OLS estimate anyway, so bagging will accomplish nothing but introduce Monte Carlo error. We expect to get something qualitatively different from our data when the underlying estimator is unstable. So bagging tends to be applied with highly unstable methods (like CART) and not with highly stable methods (like logistic regression or LDA). $\endgroup$ – guy Sep 5 '18 at 3:24
  • 1
    $\begingroup$ Random forests are actually usually superior to bagged trees, as, not only is bagging occurring, but random selection of a subset of features at every node is occurring, and, in practice, this reduces the correlation between trees, which improves the effectiveness of the final averaging step. $\endgroup$ – jbowman Sep 5 '18 at 3:30

I find this a great question. As mentioned by @guy in the comments section, bagging is more useful when the estimator / training algorithm is highly unstable and, I would add to that, non-linear. OLS or MLE are exact methods, on the same dataset they will always converge to the same solution.

CART on the other hand solves an NP-hard problem via greedy search and will produce different results even on the same dataset. There is a significant added overfitting/variance in CART's predictions and this is what bagging makes use of.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.