14
$\begingroup$

For boosting algorithms, I would say that they evolved pretty well. In early 1995 AdaBoost was introduced, then after some time it was Gradient Boosting Machine (GBM). Recently, around 2015 XGBoost was introduced, which is accurate, handles overfitting and has become a winner of multiple Kaggle competitions. In 2017 LightGBM was introduced by Microsoft, it offers a significantly lower training time comparing to XGBoost. Also, CatBoost was introduced by Yandex for handling categorical features.

Random Forest was introduced in early 2000s, but has there been any worthy successors to it? I think if a better bagging algorithm than Random Forest existed (which can be easily applied in practice) it would have gained some attention at places like Kaggle. Also, why did boosting became the more popular ensemble technique, is it because you can build less trees for an optimal prediction?

$\endgroup$
  • 1
    $\begingroup$ adaBoost was actually introduced in 1995, but that's a minor point that doesn't alter your fundamental thesis. $\endgroup$ – jbowman May 9 '18 at 21:52
  • 3
    $\begingroup$ Since random forests we've also seen the introduction of extremely randomized trees, although I'm not really aware of any good evidence that these outperform random forests with any consistency, so they may not be a "worthy" successor... $\endgroup$ – Jake Westfall May 10 '18 at 16:27
  • 1
    $\begingroup$ BART (arxiv.org/abs/0806.3286) is a Bayesian model which evolved from the single tree Bayesian CART and is inspired by the classical ensemble methods. It's worth exploring. $\endgroup$ – Zen May 10 '18 at 16:43
  • $\begingroup$ boosting became more popular since it handles many problems successfully with weak learner techniques $\endgroup$ – Refael May 22 '18 at 16:47
  • $\begingroup$ Regularized greedy forests could be worth mentioning (slow but some good results) and quantile random forests for their cool side effects. $\endgroup$ – Michael M May 22 '18 at 17:32
3
$\begingroup$

xgboost, catboost and lightgbm use some features of random forest (random sampling of variables/observations), so I think they are a successor of boosting and RF together and take the best things from both. ;)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.