The general idea of boosted decision trees is to use very simple trees in the following manner (simplified, for intuition only):

  1. start with a simple tree,
  2. fit another simple tree on the residuals,
  3. find optimal coefficient and add it to the model,
  4. do 2. and 3. several times until some stopping criterion is met.

In a way this is similar to how a "deep" decision tree itself is built, the difference being that the optimization for boosting is global whereas the deeper levels of the tree are being optimized locally on the remaining groups (greedy learner).

As an example here gbm's accuracy is comparable to C5.0 (gbm being even slightly less accurate):

My questions
First of all: are the above observations so far correct?
Second: in which situations are deep decision trees performing better (e.g. accuracy-wise) than boosted simple trees?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.