# XGBoost - Can we find a “better” objective function than RMSE for regression?

If we think back to linear models for a moment, we have Ordinary Least Squares (OLS) versus Generalized Linear Models (GLM). Without going too in-depth, it can be said that GLMs "improve" upon OLS by relaxing some of the assumptions, making it more robust to different types of data. The underlying training algorithm is also somewhat different; OLS minimizes the root mean squared error (RMSE) while GLMs minimize deviance. (I realize that RMSE is a special case of deviance). This allows us to build linear models based on, say, the gamma distribution, inverse gaussian, etc.

My question is: does the same logic hold true for gradient boosted trees? Since we're working with tree based algorithms now, I'd think that it's not subject to the same assumptions/distributional restrictions as linear models. In the XGBoost package, for example, the default objective function for regression is RMSE. You can define a custom objective if you wish, but does it matter? Does it make sense to do so?

In other words, can we possibly improve our predictive power by setting XGBoost to minimize deviance (say, of a gamma distribution) versus RMSE?

• I'm not sure I understand your distinction between RMSE and deviance. Deviance in minimized at the MLE, so the MLE of a GLM gives you minimal deviance. The difference in robustness comes from the setup of a GLM, which allows for variance that depends on expectation (through the link function), as opposed to OLS where the variance is assumed to be the same. To me it sounds like you want to use tree models to estimate expectation and variance, where your tree takes place of the link function. – Alex R. Jun 8 '16 at 19:36
• @Alex R.: Let's say I run an OLS, and then a GLM with a gamma distribution. If there is no link function for the GLM (identity link), are you saying that the two will result in identical estimates? I don't think so, since OLS is equivalent to a GLM with a normal distribution, not a gamma distribution. – AdmiralWen Jun 8 '16 at 21:16
• With a gamma distribution of course they are not the same. OLS is equivalent to MLE for gaussian errors. The "improvement" on OLS is to use MLE of a different form. – Alex R. Jun 8 '16 at 21:28
• @Alex R.: That's what I was trying to ask - about the usage of different distributions/forms. I guess I wasn't clear in my question - will make the edit. What if we set the XGBoost objective to minimize the deviance function of a gamma distribution, instead of minimize RMSE? Are there conceivable scenarios where you'd want to do that? (Or if not gamma deviance, what other objectives might you minimize for a regression problem?) – AdmiralWen Jun 8 '16 at 21:56
• Gini coefficient perhaps? – Firebug Jun 8 '16 at 22:08