5
$\begingroup$

My understaning is that boosting is a method by which you have several weak models trained in sequence. Each is trained on the full trainig data, but with greater emphasis placed on the weaknesses of the previously trained model.

I have several questions about the weak models that go into a boosted model. Please tell me if they are different enough that they should be separated into new questions.

  1. Do the weak models all need to be of the same kind? Could I follow a decision stump with a small neural network, for instance?

  2. Why not train a more complex model sequentially the same way - find where it was wrong, and put greater emphasis on what it got wrong? If it's speed, why is training lots of small models so much faster than one larger model? Related

  3. How are the features and architecture for a weak model chosen? I'd guess that they'd be random and identical, respectively, but am unsure how that works with their ability to work as feature extraction algorithms. I ask about that further here.

Thanks in advance!

$\endgroup$
2
  • $\begingroup$ good question +1. here is my related answer. stats.stackexchange.com/questions/253789/… $\endgroup$ – Haitao Du Jun 25 '20 at 5:32
  • $\begingroup$ To answer the second question, boosting is, in fact, training a complex model sequentially. In particular, boosting is nothing more than gradient descent performed in the space of functions, where at each time-step the gradient is approximated by fitting a weak learner. For more information and references see stats.stackexchange.com/questions/472766/… $\endgroup$ – user123098123 Jun 25 '20 at 6:56
5
$\begingroup$
  1. Do the weak models all need to be of the same kind? Could I follow a decision stump with a small neural network, for instance?

They do not have to be all of the same type, but there are good reasons for using simple models that train quickly, see below.

  1. Why not train a more complex model sequentially the same way - find where it was wrong, and put greater emphasis on what it got wrong? If it's speed, why is training lots of small models so much faster than one larger model?

It's not speed.

You can try your idea out with any of the standard boosting libraries: just set the tree depth to a very large number. You'll find that the model overfits quite quickly, and applying early stopping to minimize a test set loss will lead to worse predictions than boosting weak models more times.

The underlying philosophy of boosting is: make small adjustments to the prediction function gradually, evolving its shape in a slow and controlled manner to combat overfitting. You want building the complex predictive function to be the job of the boosting, not the of the weak learner being boosted.

This makes sense from a bias-variance perspective. Boosting strong models will have lower bias at a small number of boosting stages, but you pay a price in variance, since complex models tend to have higher variance. Boosting weak models slowly makes small adjustments to the predictions, which controls the model variance, but you have to boost longer to make up for the deficit in bias. Eventually you do make up this initial bias deficit, which is the power of boosting.

  1. How are the features and architecture for a weak model chosen? I'd guess that they'd be random and identical, respectively, but am unsure how that works with their ability to work as feature extraction algorithms.

As argued, you generally want a pretty simple model for this part, so there's not too many options. Some considerations:

  • Boosting linear functions leads to a linear output, so you want weak learners that can capture non-linearities.
  • Interactions between features are also a nice thing to have, so the weak learner should be able to express interactions.

These pretty directly lead towards using shallow decision trees as weak learners, which is standard.

  1. How do you or the algorithm know when to stop making new weak learners?

You monitor the test (or cross validation) error. As you boost, it will initially decrease, then at some point will flatten and then increase. As usual, you take the number of boosting stages that minimizes the test error.

$\endgroup$
2
  • $\begingroup$ Super helpful, thanks! How do you or the algorithm know when to stop making new weak learners? I'd think that if this continued on sufficiently long, you'd run into the same problem of overfitting. Also, a minor thing - could you put the question numbers on your individual answers to make it easier to reference? $\endgroup$ – fyzx92 Jun 25 '20 at 16:07
  • $\begingroup$ @fyzx92 I edited in an answer to your question. $\endgroup$ – Matthew Drury Jun 25 '20 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.