I thought it would be interesting to talk about two of the best ensemble methods off-the-shelf: Random Forests and Boosting.

  • When would you apply one method rather than the other one?

I would use whichever one performed better out of sample.

So far, I've found in impossible to tell which model will be better for a novel problem a priori.


Just to start, a quick thought.

Random Forest can run in parallel and they are much faster to train, Boosting is an iterative algorithm instead. However, Boosting might converge early iteration-wise.

Boosting might overfit when there are many noisy features but Random Forest also does.

On the other hand, their target is almost similar: produce many different weak learners as much different as possible from each others. Random Forests tackle the problem with randomization and Boosting focuses on mis-classified examples of previous models to build a different one.

  • $\begingroup$ Random forest doesn't try to produce many "different weak" learners. The learners are random but not weak in the sense that they are just slightly better than random guess. $\endgroup$ – SmallChess Sep 17 '16 at 0:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.