I want to run gradient boosting regression on a dataset whose rows are not independent. Specifically, the rows are clustered, and you could consider the clustering variable to be a random effect.

  1. What is the effect of ignoring the random effect, i.e. simply running the classifier on the target and the other features?
  2. What open source packages are available that can account for clustered data for gradient boosting?
  3. Any caveats to using the procedures from 2?

Edit: I saw How can I include random effects into a randomForest. I will now restrict my question to GBMs.

  • 1
    $\begingroup$ So basically time series / repeated measures, this is not going to fly with RF, which does not take into account the nature of the data when bootstrapping data randomly for each tree. Choose a different model. $\endgroup$ – user2974951 Jul 19 '19 at 8:16
  • $\begingroup$ Is there a specific reason for not using a mixed effects model ? $\endgroup$ – Robert Long Jul 19 '19 at 9:16
  • $\begingroup$ @RobertLong I suppose predictive power of tree ensembles is usually better than linear models. Plus, another property of the tree models that I need for my particular problem is that they don't interpolate between points, which linear models do. $\endgroup$ – ved Jul 19 '19 at 16:26
  • $\begingroup$ Why do you think interpolation is impossible for tree-based models? $\endgroup$ – mkt - Reinstate Monica Jul 19 '19 at 16:33
  • $\begingroup$ @mkt tree models are locally constant in the "box" defined by cut points on the features. Two "boxes" that are next to each other can have totally different values. $\endgroup$ – ved Jul 19 '19 at 16:42

Gradient boosting with random effects was developed by Patrick Miller and described in Miller, McArtor, & Lubke (2017). Unfortunately that reference is just an abstract of a poster, but there appears to be a related arxiv paper here. It refers to an R package, metboost, which doesn't exist. Instead, there is the package mvtboost, written by Miller. The CRAN version of mvtboost doesn't have this functionality, but the GitHub version does through the funciton metb. This only works for continuous outcomes, and there doesn't seem to be nay plan to add binary outcomes as a feature.


Not the answer you're looking for? Browse other questions tagged or ask your own question.