29
$\begingroup$

I'm not even sure that the question makes much sense, but I think I saw a couple of titles of papers where they proposed random forest with random effects. Is this possible in R?

$\endgroup$
4
  • 1
    $\begingroup$ Yes, It doesn't make much sense. What do you mean by random effects? $\endgroup$
    – Simone
    Jun 18 '14 at 3:44
  • 2
    $\begingroup$ I am thinking of something similar to what you can do with the lmer function where you can include a random effect as (1| effect). $\endgroup$
    – mguzmann
    Jun 18 '14 at 5:26
  • $\begingroup$ So this is simulated annealing on a random forest? econpapers.repec.org/article/bpjjqsprt/… researchgate.net/publication/… $\endgroup$ Jul 24 '14 at 21:54
  • 2
    $\begingroup$ I'm not quite sure about what kind of randomness are the methods which you are looking at addresses. Random forests is a simple improvement over bagging by decorrelating the tree. The reason why it is called 'random' is the fact that at any instance, when a split is considered in a tree, the split candidate is chosen from a random subset m of say p predictors. Usually, m ~ sqrt(p). And every time a split happens, a random subset of predictors is chosen hence, random forest. $\endgroup$
    – psteelk
    Jul 30 '14 at 3:21
3
$\begingroup$

Currently, this paper (doi:10.1177/0962280220946080) does a revision of previous algorithms, including those cited in previous answers. Further, that paper introduce the R library LongituRF, which allows to compute all those algorithms and the new ones.

$\endgroup$
4
  • $\begingroup$ Has anyone tried this package? i've struggling with how to specify the Z argument of the REEMforest() function. $\endgroup$ Apr 6 at 14:26
  • 1
    $\begingroup$ I've tried. As the unique I want as random is the intercept, my Z is a matrix with all 1 of 1 column and the same rows as Y or X. $\endgroup$
    – iago
    Apr 23 at 8:05
  • $\begingroup$ Getting lots of "Error in solve.default(S1) : Lapack routine dgesv: system is exactly singular:" every time i use it with real data (tried different studies and different specifications of Z). This is apparently common see stackoverflow.com/questions/66188123/… $\endgroup$ Apr 26 at 14:03
  • 1
    $\begingroup$ In fact, I also got that. I only got results with the other functions of the package. Thank you for the link! $\endgroup$
    – iago
    Apr 26 at 18:39
17
+25
$\begingroup$

Yeah it's possible. You should check out "RE-EM Trees: A Data Mining Approach for Longitudinal and Clustered Data," and the associated R package REEMtree.

It's been a while since I looked at the paper. I recall the authors had not yet tried forming ensembles of these trees, but that nothing suggested it wouldn't work.

$\endgroup$
3
  • 1
    $\begingroup$ REEMtree is not random effects applied to random forests. It is applied to recursive partitioning, which is only part of what goes into a random forest model. So I don't think this answer deserves a higher score than Bill Denney's. Unfortunately, my upvote on it is locked. $\endgroup$ Jul 31 '14 at 22:05
  • 2
    $\begingroup$ Come on, once you've got the tree, how hard is it to build the forest? And you're welcome. $\endgroup$
    – Ben Ogorek
    Aug 1 '14 at 3:50
  • 1
    $\begingroup$ Well, seeing as how random forest adds on bootstrap sampling, tuning the number of randomly chosen features to try, aggregation of the tree results, etc, and we need a random effect on the random forest predictions, not the predictions of individual trees in that forest, augmenting REEMtree is not as good a solution as reading the article Bill cited and requesting the R code from its authors. $\endgroup$ Aug 1 '14 at 15:39
16
$\begingroup$

They are not commonly used together, and care should be taken before combining them.

Random forests are typically used as classifiers. The reason that you would use a random forest instead of another method (e.g. K-means clustering) is that you may have a large number of dimensions that you want to classify by. The issue with having the large number of dimensions is that if you wanted to test all combinations of dimension orders, you would have a large number of choices (it grows faster than the number of dimensions factorial).

Random effects are typically used in regression with repeated measures of the same thing. They are commonly used in mixed effects models where the term mixed refers to both fixed and random effects. The fixed effects are thought to represent the parameters that you will see again (e.g. a drug or a person's age). The random effects are thought to represent an instance of variability around a parameter that you will not see again (e.g. a specific person).

There are examples using them together when there is clustered data http://dx.doi.org/10.1080/00949655.2012.741599 and http://www2.ims.nus.edu.sg/Programs/014swclass/files/denis.pdf.

I'm unaware of any R packages that can do this analysis.

$\endgroup$
3
  • 4
    $\begingroup$ In addition, the authors of this work are happy to share the R code of their implementation with you. Just email them. It's what I did. $\endgroup$ Jul 31 '14 at 22:06
  • $\begingroup$ I contacted Larocque, who contacted Hajjam, who emailed me within a couple of days. $\endgroup$ Aug 7 '14 at 4:06
  • 3
    $\begingroup$ Fair warning, though, the R code available only implements random forest for continuous data. You'll need to extend it to deal with categorical data. $\endgroup$ Aug 7 '14 at 4:07
16
$\begingroup$

Mixed Effects Random Forests (MERFs) are a thing. As the answer above states, there's some great research about them by Dr. Larocque's group at HEC Montreal. The paper is here: http://www.tandfonline.com/doi/abs/10.1080/00949655.2012.741599.

Essentially it is a theoretically sound way to combine the non-linear modeling of random forests with linear random effects.

We just released an open source package in Python implementing MERF using the above algorithm in the paper.

We wrote a detailed blog post about the package and how to use it for clustered data sets.

$\endgroup$
1
  • 7
    $\begingroup$ any thoughts to implementing this in R or adding partial dependency plot functionality $\endgroup$ Feb 5 '19 at 17:35
2
$\begingroup$

Instead of random forest, you can also use tree-boosting for the fixed effects part in a model with random effects. The GPBoost library with Python and R packages builds on LightGBM and allows for combining tree-boosting and mixed effects models. Simply speaking it is an extension of linear mixed effects models where the fixed-effects are learned using tree-boosting. See this blog post and Sigrist (2020) for further information.

Disclaimer: I am the author of the GPBoost library.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.