If we think back to linear models for a moment, we have Ordinary Least Squares (OLS) versus Generalized Linear Models (GLM). Without going too in-depth, it can be said that GLMs "improve" upon OLS by relaxing some of the assumptions, making it more robust to different types of data. The underlying training algorithm is also somewhat different; OLS minimizes the root mean squared error (RMSE) while GLMs minimize deviance. (I realize that RMSE is a special case of deviance). This allows us to build linear models based on, say, the gamma distribution, inverse gaussian, etc.

My question is: does the same logic hold true for gradient boosted trees? Since we're working with tree based algorithms now, I'd think that it's not subject to the same assumptions/distributional restrictions as linear models. In the XGBoost package, for example, the default objective function for regression is RMSE. You can define a custom objective if you wish, but does it matter? Does it make sense to do so?

In other words, can we possibly improve our predictive power by setting XGBoost to minimize deviance (say, of a gamma distribution) versus RMSE?

  • $\begingroup$ I'm not sure I understand your distinction between RMSE and deviance. Deviance in minimized at the MLE, so the MLE of a GLM gives you minimal deviance. The difference in robustness comes from the setup of a GLM, which allows for variance that depends on expectation (through the link function), as opposed to OLS where the variance is assumed to be the same. To me it sounds like you want to use tree models to estimate expectation and variance, where your tree takes place of the link function. $\endgroup$ – Alex R. Jun 8 '16 at 19:36
  • $\begingroup$ @Alex R.: Let's say I run an OLS, and then a GLM with a gamma distribution. If there is no link function for the GLM (identity link), are you saying that the two will result in identical estimates? I don't think so, since OLS is equivalent to a GLM with a normal distribution, not a gamma distribution. $\endgroup$ – AdmiralWen Jun 8 '16 at 21:16
  • $\begingroup$ With a gamma distribution of course they are not the same. OLS is equivalent to MLE for gaussian errors. The "improvement" on OLS is to use MLE of a different form. $\endgroup$ – Alex R. Jun 8 '16 at 21:28
  • $\begingroup$ @Alex R.: That's what I was trying to ask - about the usage of different distributions/forms. I guess I wasn't clear in my question - will make the edit. What if we set the XGBoost objective to minimize the deviance function of a gamma distribution, instead of minimize RMSE? Are there conceivable scenarios where you'd want to do that? (Or if not gamma deviance, what other objectives might you minimize for a regression problem?) $\endgroup$ – AdmiralWen Jun 8 '16 at 21:56
  • $\begingroup$ Gini coefficient perhaps? $\endgroup$ – Firebug Jun 8 '16 at 22:08

To answer your question you need to define what is "better". If your goal is to reach a smaller distance to a gamma distributed variable measured in squared distance than you should build your objective function in that way. A very common problem is to find a solution that minimizes Mean Absolute Error. That cost function is different from RMSE and using a different objective function can help in minimizing the MAE.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.