How does boosting work?

Question

What is the easiest way to understand boosting?

Why doesn't it boost very weak classifiers "to infinity" (perfection)?

Answer 1

In plain English: If your classifier misclassifies some data, train another copy of it mainly on this misclassified part with hope that it will discover something subtle. And then, as usual, iterate. On the way there are some voting schemes that allow to combine all those classifiers' predictions in sensible way.

Because sometimes it is impossible (the noise is just hiding some of the information, or it is not even present in the data); on the other hand, boosting too much may lead to overfitting.

Answer 2

Boosting employs shrinkage through the learning rate parameter, which, coupled with k-fold cross validation, "out-of-bag" (OOB) predictions or independent test set, determine the number of trees one should keep in the ensemble.

We want a model that learns slowly, hence there is a trade-off in terms of the complexity of each individual model and the number of models to include. The guidance I have seen suggests you should set the learning rate as low as is feasibly possible (given compute time and storage space requirements), whilst the complexity of each tree should be selected on basis of whether interactions are allowed, and to what degree, the more complex the tree, the more complex the interactions that can be represented.

The learning rate is chosen in the range $[0,1]$. Smaller values ($<0.01$) preferred. This is a weighting applied to each tree to down weight the contribution of each model to the fitted values.

k-fold CV (or OOB predictions or independent test set) is used to decide when the boosted model has started to overfit. Essentially it is this that stops us boosting to the perfect model, but it is better to learn slowly so we have a large ensemble of models contributing to the fitted model.