What am I missing here?

I don't think you're really missing anything!

Another observation is that a sum of subsequent linear regression models can be represented as a single regression model as well (adding all intercepts and corresponding coefficients) so I cannot imagine how that could ever improve the model. The last observation is that a linear regression (the most typical approach) is using sum of squared residuals as a loss function - the same one that GB is using.

Seems to me that you nailed it right there, and gave a short sketch of a proof that linear regression just beats boosting linear regressions in this setting.

To be pedantic, both methods are attempting to solve the following optimization problem

$$ \hat \beta = \text{argmin}_\beta (y - X \beta)^t (y - X \beta) $$

Linear regression just observes that you can solve it directly, by finding the solution to the linear equation

$$ X^t X \beta = X^t y $$

This automatically gives you the best possible value of $\beta$ out of all possibilities.

Boosting, whether your weak classifier is a one variable or multi variable regression, gives you a sequence of coefficient vectors $\beta_1, \beta_2, \ldots$. The final model prediction is, as you observe, a sum, and has the same functional form as the full linear regressor

$$ \beta_1 X + \beta_2 X + \cdots + \beta_n X = (\beta_1 + \beta_2 + \cdots + \beta_n) X $$

Each of these steps is chosen to further decrease the sum of squared errors. But we could have found the minimum possible sum of square errors within this functional form by just performing a full linear regression to begin with.

A possible defense of boosting in this situation could be the implicit regularization it provides. Possibly (I haven't played with this) you could use the early stopping feature of a gradient booster, along with a cross validation, to stop short of the full linear regression. This would provide a regularization to your regression, and possibly help with overfitting. This is not particularly practical, as one has very efficient and well understood options like ridge regression and the elastic net in this setting.

Boosting shines when there is no terse functional form around. Boosting decision trees lets the functional form of the regressor/classifier evolve slowly to fit the data, often resulting in complex shapes one could not have dreamed up by hand and eye. When a simple functional form is desired, boosting is not going to help you find it (or at least is probably a rather inefficient way to find it).

What am I missing here?

I don't think you're really missing anything!

Another observation is that a sum of subsequent linear regression models can be represented as a single regression model as well (adding all intercepts and corresponding coefficients) so I cannot imagine how that could ever improve the model. The last observation is that a linear regression (the most typical approach) is using sum of squared residuals as a loss function - the same one that GB is using.

Seems to me that you nailed it right there, and gave a short sketch of a proof that linear regression just beats boosting linear regressions in this setting.

To be pedantic, both methods are attempting to solve the following optimization problem

$$ \hat \beta = \text{argmin}_\beta (y - X \beta)^t (y - X \beta) $$

Linear regression just observes that you can solve it directly, by finding the solution to the linear equation

$$ X^t X \beta = X^t y $$

This automatically gives you the best possible value of $\beta$ out of all possibilities.

Boosting, whether your weak classifier is a one variable or multi variable regression, gives you a sequence of coefficient vectors $\beta_1, \beta_2, \ldots$. The final model prediction is, as you observe, a sum, and has the same functional form as the full linear regressor

$$ X \beta_1 + X \beta_2 + \cdots + X \beta_n = X (\beta_1 + \beta_2 + \cdots + \beta_n) $$

Each of these steps is chosen to further decrease the sum of squared errors. But we could have found the minimum possible sum of square errors within this functional form by just performing a full linear regression to begin with.

A possible defense of boosting in this situation could be the implicit regularization it provides. Possibly (I haven't played with this) you could use the early stopping feature of a gradient booster, along with a cross validation, to stop short of the full linear regression. This would provide a regularization to your regression, and possibly help with overfitting. This is not particularly practical, as one has very efficient and well understood options like ridge regression and the elastic net in this setting.

Boosting shines when there is no terse functional form around. Boosting decision trees lets the functional form of the regressor/classifier evolve slowly to fit the data, often resulting in complex shapes one could not have dreamed up by hand and eye. When a simple functional form is desired, boosting is not going to help you find it (or at least is probably a rather inefficient way to find it).

What am I missing here?

I don't think you're really missing anything!

Another observation is that a sum of subsequent linear regression models can be represented as a single regression model as well (adding all intercepts and corresponding coefficients) so I cannot imagine how that could ever improve the model. The last observation is that a linear regression (the most typical approach) is using sum of squared residuals as a loss function - the same one that GB is using.

Seems to me that you nailed it right there, and gave a short sketch of a proof that linear regression just beats boosting linear regressions in this setting.

To be pedantic, both methods are attempting to solve the following optimization problem

$$ \hat \beta = \text{argmin}_\beta (y - X \beta)^t (y - X \beta) $$

Linear regression just observes that you can solve it directly, by finding the solution to the linear equation

$$ X^t X \beta = X^t y $$

This automatically gives you the best possible value of $\beta$ out of all possibilities.

Boosting, whether your weak classifier is a one variable or multi variable regression, gives you a sequence of coefficient vectors $\beta_1, \beta_2, \ldots$. The final model prediction is, as you observe, a sum, and has the same functional form as the full linear regressor

$$ \beta_1 X + \beta_2 X + \cdots + \beta_n X = (\beta_1 + \beta_2 + \cdots + \beta_n) X $$

Each of these steps is chosen to further decrease the sum of squared errors. But we could have found the minimum possible sum of square errors within this functional form by just performing a full linear regression to begin with. So, as you rightly suspect, boosting adds nothing.

Boosting shines when there is no terse functional form around. Boosting decision trees lets the functional form of the regressor/classifier evolve slowly to fit the data, often resulting in complex shapes one could not have dreamed up by hand and eye. When a simple functional form is desired, boosting is not going to help you find it (or at least is probably a rather inefficient way to find it).

What am I missing here?

I don't think you're really missing anything!

Another observation is that a sum of subsequent linear regression models can be represented as a single regression model as well (adding all intercepts and corresponding coefficients) so I cannot imagine how that could ever improve the model. The last observation is that a linear regression (the most typical approach) is using sum of squared residuals as a loss function - the same one that GB is using.

Seems to me that you nailed it right there, and gave a short sketch of a proof that linear regression just beats boosting linear regressions in this setting.

To be pedantic, both methods are attempting to solve the following optimization problem

$$ \hat \beta = \text{argmin}_\beta (y - X \beta)^t (y - X \beta) $$

Linear regression just observes that you can solve it directly, by finding the solution to the linear equation

$$ X^t X \beta = X^t y $$

This automatically gives you the best possible value of $\beta$ out of all possibilities.

Boosting, whether your weak classifier is a one variable or multi variable regression, gives you a sequence of coefficient vectors $\beta_1, \beta_2, \ldots$. The final model prediction is, as you observe, a sum, and has the same functional form as the full linear regressor

$$ \beta_1 X + \beta_2 X + \cdots + \beta_n X = (\beta_1 + \beta_2 + \cdots + \beta_n) X $$

Each of these steps is chosen to further decrease the sum of squared errors. But we could have found the minimum possible sum of square errors within this functional form by just performing a full linear regression to begin with.

A possible defense of boosting in this situation could be the implicit regularization it provides. Possibly (I haven't played with this) you could use the early stopping feature of a gradient booster, along with a cross validation, to stop short of the full linear regression. This would provide a regularization to your regression, and possibly help with overfitting. This is not particularly practical, as one has very efficient and well understood options like ridge regression and the elastic net in this setting.

Boosting shines when there is no terse functional form around. Boosting decision trees lets the functional form of the regressor/classifier evolve slowly to fit the data, often resulting in complex shapes one could not have dreamed up by hand and eye. When a simple functional form is desired, boosting is not going to help you find it (or at least is probably a rather inefficient way to find it).