In my Master thesis i compare ols regression to regression tree to predict wages.

I thought that i will get better prediction with the regression tree because it cathes more interactions.

But now i get smaller out of sample mse for my ols modell compared to the regression tree model.

How this could be ?


A regression tree is more complicated than a simple regression (assuming the same model).

Complex models do not necessarily outperform simple ones. A simple misspecified model may yield better predictions than a more complex correctly specified one because of the bias-variance tradeoff.

And if your original simple OLS model is already correctly specified, then adding more flexibility by putting it into a regression tree will not reduce the bias (if the OLS model is correctly specified, there is no bias), but it will increase the variance of your parameter estimates, and therefore also of your predictions.

  • $\begingroup$ Thank your for the fast answer. I understand what you mean. But one thing is not clear for me. For my regression tree i run a 10 fold cross validation and get the result i described above. Now i tried a 5 fold cross validation and now the mse of the regression tree is lower compared to the ols mse. How this result can be explained? $\endgroup$ – MasterStudent1992 Apr 18 at 12:06
  • $\begingroup$ Hm. The MSE should not differ all that much between models chosen by 5 fold or 10 fold CV. How stable are the models your CV chooses? Have you tried re-running each CV multiple times and seeing how stable the models or predictions are in each case? $\endgroup$ – Stephan Kolassa Apr 18 at 14:05
  • $\begingroup$ it was code mistake with the results for 10 and 5 k fold cross validation. I accepted your answer because it was helpful. I tried now 100 different seeds and computed 100 models for regression and for ols and computed the mse values for each models and avaregd the mse values over all 100 predictions. The mse values of the regression tree averaged over the 100 predictions is now smaller as the ols mse averaged over the 100 predictions. Is this a good approach to show that regression tree performs better on average ? $\endgroup$ – MasterStudent1992 Apr 18 at 22:01
  • $\begingroup$ That sounds good. $\endgroup$ – Stephan Kolassa Apr 19 at 10:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.