19
$\begingroup$

Can Friedman's gradient boosting machine achieve better performance than Breiman's Random Forest? If so, in which conditions or what kind of data set can make gbm better?

$\endgroup$
5
  • 6
    $\begingroup$ There is no way to tell a priori; you have to try it out. $\endgroup$
    – bayerj
    Jul 25, 2013 at 14:16
  • $\begingroup$ well, in practice, Boosting almost always outperform RF... Though I don't really know exactly why, I have personally not encountered any instance where RF outperformed Boosting. $\endgroup$
    – Antoine
    Oct 1, 2015 at 19:10
  • $\begingroup$ @Antoine learning with unlabeled data and/or label noise is one particularly terrible use-case for boosting. $\endgroup$ Oct 2, 2015 at 8:37
  • $\begingroup$ Well, RF and Boosting are primarily used for supervised learning tasks, even if sometimes it is true that RF can be used for clustering. Adaboost is not very robust to mislabeling because of the exponential loss function which is highly influenced by noise, but Stochastic Gradient Boosting in the general case (with multinomial deviance for instance) is more robust. $\endgroup$
    – Antoine
    Oct 2, 2015 at 8:47
  • $\begingroup$ @MarcClaesen could you have a look at this question please? $\endgroup$
    – Antoine
    Oct 3, 2015 at 9:02

2 Answers 2

13
$\begingroup$

The following provides an explanation as per why Boosting generally outperforms Random Forest in practice, but I would be very interested to know which other different factors may explain Boosting's edge over RF in specific settings.

Basically, within the $error=bias+variance$ framework, RF can only reduce error through reducing the variance (Hastie et al. 2009 p. 588). The bias is fixed and equal to the bias of a single tree in the forest (hence the need to grow very large trees, that have very low bias).

On the other hand, Boosting reduces bias (by adding each new tree in the sequence so that what was missed by the preceding tree is captured), but also variance (by combining many models).

So, Boosting reduces error on both fronts, whereas RF can only reduce error through reducing variance. Of course, as I said, there might be other explanations for the better performance of Boosting observed in practice. For instance, page 591 of the aforementioned book, it is said that Boosting outperforms RF on the nested sphere problem because in that particular case the true decision boundary is additive. (?) They also report that Boosting does better than RF for the spam and the California housing data.

Another reference that found Boosting to outperform RF is Caruana and Niculescu-Mizil 2006. Unfortunately, they report the results but don't try to explain what causes them. They compared the two classifiers (and many more) on 11 binary classification problems for 8 different performance metrics.

$\endgroup$
7
$\begingroup$

As bayerj said it, there is no way to know a priori !

Random Forests are relatively easy to calibrate: default parameters of most implementations (R or Python, per example) achieve great results.

On the other hand, GBMs are hard to tune (a too large number of tree leads to overfit, maximum depth is critical, the learning rate and the number of trees act together...) and longer to train (multithreaded implementations are scarce). A loosely performed tuning may lead to low performance.

However, from my experience, if you spend enough time on GBMs, you are likely to achieve better performance than random forest.

Edit. Why do GBMs outperform Random Forests? Antoine's answer is much more rigorous, this is just an intuitive explanation. They have more critical parameters. Just like the random forests, you can calibrate the number of trees and $m$ the number of variables on which trees are grown. But you can also calibrate the learning rate and the maximum depth. As you observe more different models than you would do with a random forest, you are more likely to find something better.

$\endgroup$
1
  • 1
    $\begingroup$ A loosely performed tuning may lead to dramatic performance? Beware of misinterpretation, because in English dramatic means very good, exceptional, phenomenal, etc.! I guess that's the opposite of what you wanted to say... In addition, do you have any explanation as per why carefully tuned GBMs outperform RF? This is basically the question... $\endgroup$
    – Antoine
    Oct 1, 2015 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.