I am confused when I read the boosting papers.

In boosting our model is a sum of base learners:

$$ f(x)=\sum_{m=1}^M b_m(x) $$

where $M$ is number of iterations in boosting, $b_m$ is the model for $m^{th}$ iteration.

Now, here is my question: can the base learner be linear? If the base learner is linear, does the whole model reduced into a simpler linear model?

For example, suppose we just run $2$ iterations, and $b_1=\beta_0+ \beta_1x$ and $b_2=\theta_0+ \theta_1x$, then

$$ f(x)=\sum_{m=1}^2 b_m(x)=\beta_0+ \beta_1x+\theta_0+ \theta_1x=(\beta_0+\theta_0)+ (\beta_1+ \theta_1)x $$

which is a simple linear model ! In other words, the ensemble model have the "same power" with the base learner!

What's wrong there? Or nothing wrong, we just do not want to select linear model as a base learner?

  • 3
    $\begingroup$ Your argument is correct, if you choose a linear model as your base learning, then since linear functions are closed under addition, the final model will also be linear. $\endgroup$ Aug 23 '16 at 16:00