4
$\begingroup$

I'm studying Random Forests, but after reviewing the methods I got the following line of reasoning: I feel like the big advantage of random forests happens in the bagging process where nearly uncorrelated predictions are created due to the random features, producing predictions with low variance. On the other hand, this method doesn't require your classification method to be a decision tree.

If other classification algorithms (LDA, QDA, logistic regression, etc) usually perform better than decision trees, why would I choose to use decision trees as my classifier?

Random forests seem a lot more popular than bagging with other classifiers (it even got its own name) but I don't see any particular reason why.

Improve this question
$\endgroup$
5
  • $\begingroup$ One reason might be ease of interpretation to non-ML savvy outsiders. $\endgroup$
    – gannawag
    Commented Sep 5, 2018 at 1:26
  • $\begingroup$ Doesn't the bagging process kills the interpretation? Your 'final' prediction is an average of predictions produced by possibly (and very likely) different trees. $\endgroup$
    – VFreguglia
    Commented Sep 5, 2018 at 1:30
  • $\begingroup$ Yeah good point...I was thinking that decision trees are pretty easy to understand, and it might be easier to jump from there to bagging than starting with a more complicated approach. $\endgroup$
    – gannawag
    Commented Sep 5, 2018 at 1:35
  • 1
    $\begingroup$ Ask yourself: what happens if you apply bagging to ordinary least squares? If you apply the parametric bootstrap, you will see that the center of the bootstrap distribution is just the OLS estimate anyway, so bagging will accomplish nothing but introduce Monte Carlo error. We expect to get something qualitatively different from our data when the underlying estimator is unstable. So bagging tends to be applied with highly unstable methods (like CART) and not with highly stable methods (like logistic regression or LDA). $\endgroup$
    – guy
    Commented Sep 5, 2018 at 3:24
  • 2
    $\begingroup$ Random forests are actually usually superior to bagged trees, as, not only is bagging occurring, but random selection of a subset of features at every node is occurring, and, in practice, this reduces the correlation between trees, which improves the effectiveness of the final averaging step. $\endgroup$
    – jbowman
    Commented Sep 5, 2018 at 3:30

2 Answers 2

Reset to default
1
$\begingroup$

I find this a great question. As mentioned by @guy in the comments section, bagging is more useful when the estimator / training algorithm is highly unstable and, I would add to that, non-linear. OLS or MLE are exact methods, on the same dataset they will always converge to the same solution.

CART on the other hand solves an NP-hard problem via greedy search and will produce different results even on the same dataset. There is a significant added overfitting/variance in CART's predictions and this is what bagging makes use of.

Improve this answer
$\endgroup$
1
$\begingroup$

random selection of feature-subset is used at each node in Random Forest RF which improves variance by reducing correlation between trees(ie: it uses both features and row data randomly)

While Bagging improves variance by averaging/majority selection of outcome from multiple fully grown trees on variants of training set. It uses Bootstrap with replacement to generate multiple training sets.(only Row data is used here)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.