Suppose, we have a normal linear model given regressor matrix $X$, can one expect a boosted linear model with the same regressor matrix to be have better performance?

I mean the following: I have a normal linear model trained on the train set with, say, $R^2 = 0.95$ and this model has also good predictive performance on the cross validation sets, so I believe is not overfitted. Would you expect the xgboost linear model to perform better? Personally, I don't expect this behavior.


So I tried and the CV score is even little worse. Then I thought, in principle, boosting with linear base learner is just sum of linear models, hence it is always linear model. Then it does not make much sense to me, to use linear model as a base learner.

  • $\begingroup$ $R^2 \approx 0.95$ is almost... unrealistically good. I have seen such values only in cases of severe data-leakage. Unless you are modelling a physical process recorded under experimental conditions I would be really sceptical about it. $\endgroup$
    – usεr11852
    Nov 9 '16 at 22:11
  • $\begingroup$ I don't model physical process, but I am 100% sure, there is no leakage. Of course I can run linear xgboost and see the results, however I want to know if others think there is yet a space for improvement with this high $R^2$. $\endgroup$
    – reicja
    Nov 10 '16 at 8:14
  • $\begingroup$ my answer to this post may give you answer. stats.stackexchange.com/questions/230388/… $\endgroup$
    – Haitao Du
    Nov 10 '16 at 16:32
  • $\begingroup$ OK, fair enough! Assuming that you get an $R^2 \approx 0.95$ in out-of-sample predictions (or at least repeated CV) then I wouldn't think that there is much room for improvement but if it is in-sample predictions then probably there is more to be done. @hxd1011 's link is a very good take on the matter. $\endgroup$
    – usεr11852
    Nov 10 '16 at 20:54
  • 2
    $\begingroup$ In terms of explanatory inference I agree that the in-sample $R^2$ is extremely good; nevertheless as it is in-sample it does not say anything about how generalisable is your model. As mentioned try at least some repeated CV so you can have an idea about how well it generalises with unseen data. "All models are wrong but some are useful." So the question is how would you like to use your model. $\endgroup$
    – usεr11852
    Nov 12 '16 at 12:24

I don't think this can be answered without knowing whether (i) whether the relationship is truly linear and (ii) the other assumptions of linear regression are met. If these conditions are met, I don't see how a linear regression can be improved upon. If the conditions are not met, then XGBoost (which is a boosted regression tree) could quite conceivably improve upon it. The specific $𝑅^2$ is not really relevant, for reasons discussed in this thread.


Here is the thing: Xgboost linear model will train every base model on the residual from the previous one. Which means, it tend to overfit the data. So if you use the same regressor matrix, it may not perform better than the linear regression model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.