Gradient boosting algorithm (steps) question

Question

So, far I have read following regarding boosting:

• Boosting is an ensemble technique.
• Train learner sequentially, where early learners fit simple models to the data.
• Analyze data for errors, that is, focus on records that have high errors and try to get them right.
• In the end, all the classifiers are given weights and combined to give a final prediction.

Gradient Boosting (Boosted tree) for regression: Repeatedly follow these steps:

(1) Learn a regression classifier

(2) Compute the error residual ( obs y - predicted y) per data point

(3) Learn a new model to try to predict the error residual.

So, the step I don't understand is, how do you combine step 1 and 3 ( and repeat the whole process again?) I have read that you add the two classifiers together, ( one from step 1 and second from step 3 and you repeat the process). So, my question is, how does this combining step 1 and step 3 take place and go to the next stage? Explanation with a few data points ( numerical example) to visualize the algorithm will be very helpful.

Answer 1

Step 1 and 3 correspond to the same step repeated within the iterarive loop.

Steps are:

1. Learn a model

2. Begin iteration

   • Compute errors
   • Learn a new model that can reduce previous errors

Boosting usually learn models that can reduce previous errors by two mechanisms:

• If the algorithm allows to give weights to instances, highly weight those instances corresponding to large previous errors
• Bootstrap the original sample, giving a higher probability of ocurrence to those instances corresponding to larger previous errors.