So, far I have read following regarding boosting:
- Boosting is an ensemble technique.
- Train learner sequentially, where early learners fit simple models to the data.
- Analyze data for errors, that is, focus on records that have high errors and try to get them right.
- In the end, all the classifiers are given weights and combined to give a final prediction.
Gradient Boosting (Boosted tree) for regression: Repeatedly follow these steps:
(1) Learn a regression classifier
(2) Compute the error residual ( obs y - predicted y) per data point
(3) Learn a new model to try to predict the error residual.
So, the step I don't understand is, how do you combine step 1 and 3 ( and repeat the whole process again?) I have read that you add the two classifiers together, ( one from step 1 and second from step 3 and you repeat the process). So, my question is, how does this combining step 1 and step 3 take place and go to the next stage? Explanation with a few data points ( numerical example) to visualize the algorithm will be very helpful.