I want to run gradient boosting regression on a dataset whose rows are not independent. Specifically, the rows are clustered, and you could consider the clustering variable to be a random effect.

  1. What is the effect of ignoring the random effect, i.e. simply running the classifier on the target and the other features?
  2. What open source packages are available that can account for clustered data for gradient boosting?
  3. Any caveats to using the procedures from 2?

Edit: I saw How can I include random effects into a randomForest. I will now restrict my question to GBMs.


marked as duplicate by Robert Long, Noah, usεr11852 says Reinstate Monic, kjetil b halvorsen, mkt - Reinstate Monica Jul 20 at 19:56

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ So basically time series / repeated measures, this is not going to fly with RF, which does not take into account the nature of the data when bootstrapping data randomly for each tree. Choose a different model. $\endgroup$ – user2974951 Jul 19 at 8:16
  • $\begingroup$ Is there a specific reason for not using a mixed effects model ? $\endgroup$ – Robert Long Jul 19 at 9:16
  • $\begingroup$ @RobertLong I suppose predictive power of tree ensembles is usually better than linear models. Plus, another property of the tree models that I need for my particular problem is that they don't interpolate between points, which linear models do. $\endgroup$ – ved Jul 19 at 16:26
  • $\begingroup$ Why do you think interpolation is impossible for tree-based models? $\endgroup$ – mkt - Reinstate Monica Jul 19 at 16:33
  • $\begingroup$ @mkt tree models are locally constant in the "box" defined by cut points on the features. Two "boxes" that are next to each other can have totally different values. $\endgroup$ – ved Jul 19 at 16:42

Gradient boosting with random effects was developed by Patrick Miller and described in Miller, McArtor, & Lubke (2017). Unfortunately that reference is just an abstract of a poster, but there appears to be a related arxiv paper here. It refers to an R package, metboost, which doesn't exist. Instead, there is the package mvtboost, written by Miller. The CRAN version of mvtboost doesn't have this functionality, but the GitHub version does through the funciton metb. This only works for continuous outcomes, and there doesn't seem to be nay plan to add binary outcomes as a feature.


Not the answer you're looking for? Browse other questions tagged or ask your own question.