In Boosting/Adaboost, why the base classifier must be weak classifier?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
It is not required to have a "weak classifier" for base classifier, in theory you can even choose neural network as base classifier. However, weak classifier is recommended to avoid overfitting and better control of the model complexity.
The reason is that boosting will increase the model "variance" step by step over iterations. If the base classifier has high variance (say neural network), we may jump to far from one step to another step over iterations. The result would be it is very hard to control the model complexity. In the end, we may have a overfitting problem.
On the other hand, if we chose a simple / trivial base, such as decision stump, we can use number of iterations to control the model complexity easily, and if we increase number of iterations, we still will have a sufficient complex model.
I Would suggest you to read my other question and answers, there are many visualizations in there.
Is a decision stump a linear model?
How does linear base leaner works in boosting? And how it works in xgboost library?
Boosting: why is the learning rate called a regularization parameter?
answered Dec 30, 2016 at 5:03
-
$\begingroup$ In addition to avoiding overly complex (high-variance / overfit) models, others also suggest that weak learns in boosting help reduce computation / training time: stats.stackexchange.com/a/23390/145508 $\endgroup$ Apr 9, 2018 at 11:05