I see people often ensemble Gradient Boosted Decision Trees and Random Forests together. Does it make sense to ensemble a Decision Tree and a GBDT together? Isn't this DT already a part of GBDT?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
It is possible to create ensembles of trained machine learning models (by voting/averaging of predictions) but what is thought to be substantially better is stacking a variety of different trained machine learning algorithms together. Stacking works by training a second level machine learning model (a meta-learner) on the predictions from machine learning models that are themselves trained on the data. This is frequently recommended for small improvements in performance at the high end - stacked models have won multiple kaggle competitions.
GBMs (i.e. GBDTs) and random forests are frequently used in these, but I think they are unlikely to be used together. Why? The benefits of stacking come from including models of very different types i.e. that have learned to make accurate predictions in substantially different ways. Random forests and GBMs are structurally too similar to provide much of an advantage from including both.
This same objection applies to your idea of using both a decision tree and a GBM together, but even more strongly. And there's an additional reason not to: the performance of a single decision tree is usually so much worse than either a GBM or a random forest (hence the need for those methods) that including it in the stack would not improve performance; if anything, I would expect it to decrease it.