So, far I have read following regarding boosting:

  • Boosting is an ensemble technique.
  • Train learner sequentially, where early learners fit simple models to the data.
  • Analyze data for errors, that is, focus on records that have high errors and try to get them right.
  • In the end, all the classifiers are given weights and combined to give a final prediction.

Gradient Boosting (Boosted tree) for regression: Repeatedly follow these steps:

(1) Learn a regression classifier

(2) Compute the error residual ( obs y - predicted y) per data point

(3) Learn a new model to try to predict the error residual.

So, the step I don't understand is, how do you combine step 1 and 3 ( and repeat the whole process again?) I have read that you add the two classifiers together, ( one from step 1 and second from step 3 and you repeat the process). So, my question is, how does this combining step 1 and step 3 take place and go to the next stage? Explanation with a few data points ( numerical example) to visualize the algorithm will be very helpful.

  • $\begingroup$ I don't know if I can answer your question, but the vignettes that come with the gbm and dismo packages in R have some great information on gradient boosting and may be of help to you. $\endgroup$
    – GNG
    Dec 5, 2014 at 19:09

1 Answer 1


Step 1 and 3 correspond to the same step repeated within the iterarive loop.

Steps are:

  1. Learn a model

  2. Begin iteration

    • Compute errors
    • Learn a new model that can reduce previous errors

Boosting usually learn models that can reduce previous errors by two mechanisms:

  • If the algorithm allows to give weights to instances, highly weight those instances corresponding to large previous errors
  • Bootstrap the original sample, giving a higher probability of ocurrence to those instances corresponding to larger previous errors.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.