- Do the weak models all need to be of the same kind? Could I follow a decision stump with a small neural network, for instance?
They do not have to be all of the same type, but there are good reasons for using simple models that train quickly, see below.
- Why not train a more complex model sequentially the same way - find where it was wrong, and put greater emphasis on what it got wrong? If it's speed, why is training lots of small models so much faster than one larger model?
It's not speed.
You can try your idea out with any of the standard boosting libraries: just set the tree depth to a very large number. You'll find that the model overfits quite quickly, and applying early stopping to minimize a test set loss will lead to worse predictions than boosting weak models more times.
The underlying philosophy of boosting is: make small adjustments to the prediction function gradually, evolving its shape in a slow and controlled manner to combat overfitting. You want building the complex predictive function to be the job of the boosting, not the of the weak learner being boosted.
This makes sense from a bias-variance perspective. Boosting strong models will have lower bias at a small number of boosting stages, but you pay a price in variance, since complex models tend to have higher variance. Boosting weak models slowly makes small adjustments to the predictions, which controls the model variance, but you have to boost longer to make up for the deficit in bias. Eventually you do make up this initial bias deficit, which is the power of boosting.
- How are the features and architecture for a weak model chosen? I'd guess that they'd be random and identical, respectively, but am unsure how that works with their ability to work as feature extraction algorithms.
As argued, you generally want a pretty simple model for this part, so there's not too many options. Some considerations:
- Boosting linear functions leads to a linear output, so you want weak learners that can capture non-linearities.
- Interactions between features are also a nice thing to have, so the weak learner should be able to express interactions.
These pretty directly lead towards using shallow decision trees as weak learners, which is standard.
- How do you or the algorithm know when to stop making new weak learners?
You monitor the test (or cross validation) error. As you boost, it will initially decrease, then at some point will flatten and then increase. As usual, you take the number of boosting stages that minimizes the test error.
