22
$\begingroup$

I'm not even sure that the question makes much sense, but I think I saw a couple of titles of papers where they proposed random forest with random effects. Is this possible in R?

$\endgroup$
  • 1
    $\begingroup$ Yes, It doesn't make much sense. What do you mean by random effects? $\endgroup$ – Simone Jun 18 '14 at 3:44
  • $\begingroup$ I am thinking of something similar to what you can do with the lmer function where you can include a random effect as (1| effect). $\endgroup$ – mguzmann Jun 18 '14 at 5:26
  • $\begingroup$ So this is simulated annealing on a random forest? econpapers.repec.org/article/bpjjqsprt/… researchgate.net/publication/… $\endgroup$ – EngrStudent - Reinstate Monica Jul 24 '14 at 21:54
  • 2
    $\begingroup$ I'm not quite sure about what kind of randomness are the methods which you are looking at addresses. Random forests is a simple improvement over bagging by decorrelating the tree. The reason why it is called 'random' is the fact that at any instance, when a split is considered in a tree, the split candidate is chosen from a random subset m of say p predictors. Usually, m ~ sqrt(p). And every time a split happens, a random subset of predictors is chosen hence, random forest. $\endgroup$ – psteelk Jul 30 '14 at 3:21
13
$\begingroup$

They are not commonly used together, and care should be taken before combining them.

Random forests are typically used as classifiers. The reason that you would use a random forest instead of another method (e.g. K-means clustering) is that you may have a large number of dimensions that you want to classify by. The issue with having the large number of dimensions is that if you wanted to test all combinations of dimension orders, you would have a large number of choices (it grows faster than the number of dimensions factorial).

Random effects are typically used in regression with repeated measures of the same thing. They are commonly used in mixed effects models where the term mixed refers to both fixed and random effects. The fixed effects are thought to represent the parameters that you will see again (e.g. a drug or a person's age). The random effects are thought to represent an instance of variability around a parameter that you will not see again (e.g. a specific person).

There are examples using them together when there is clustered data http://dx.doi.org/10.1080/00949655.2012.741599 and http://www2.ims.nus.edu.sg/Programs/014swclass/files/denis.pdf.

I'm unaware of any R packages that can do this analysis.

$\endgroup$
  • 2
    $\begingroup$ In addition, the authors of this work are happy to share the R code of their implementation with you. Just email them. It's what I did. $\endgroup$ – Brash Equilibrium Jul 31 '14 at 22:06
  • $\begingroup$ I contacted Larocque, who contacted Hajjam, who emailed me within a couple of days. $\endgroup$ – Brash Equilibrium Aug 7 '14 at 4:06
  • 2
    $\begingroup$ Fair warning, though, the R code available only implements random forest for continuous data. You'll need to extend it to deal with categorical data. $\endgroup$ – Brash Equilibrium Aug 7 '14 at 4:07
10
+25
$\begingroup$

Yeah it's possible. You should check out "RE-EM Trees: A Data Mining Approach for Longitudinal and Clustered Data," and the associated R package REEMtree.

It's been a while since I looked at the paper. I recall the authors had not yet tried forming ensembles of these trees, but that nothing suggested it wouldn't work.

$\endgroup$
  • 1
    $\begingroup$ REEMtree is not random effects applied to random forests. It is applied to recursive partitioning, which is only part of what goes into a random forest model. So I don't think this answer deserves a higher score than Bill Denney's. Unfortunately, my upvote on it is locked. $\endgroup$ – Brash Equilibrium Jul 31 '14 at 22:05
  • 1
    $\begingroup$ Come on, once you've got the tree, how hard is it to build the forest? And you're welcome. $\endgroup$ – Ben Ogorek Aug 1 '14 at 3:50
  • 1
    $\begingroup$ Well, seeing as how random forest adds on bootstrap sampling, tuning the number of randomly chosen features to try, aggregation of the tree results, etc, and we need a random effect on the random forest predictions, not the predictions of individual trees in that forest, augmenting REEMtree is not as good a solution as reading the article Bill cited and requesting the R code from its authors. $\endgroup$ – Brash Equilibrium Aug 1 '14 at 15:39
9
$\begingroup$

Mixed Effects Random Forests (MERFs) are a thing. As the answer above states, there's some great research about them by Dr. Larocque's group at HEC Montreal. The paper is here: http://www.tandfonline.com/doi/abs/10.1080/00949655.2012.741599.

Essentially it is a theoretically sound way to combine the non-linear modeling of random forests with linear random effects.

We just released an open source package in Python implementing MERF using the above algorithm in the paper.

We wrote a detailed blog post about the package and how to use it for clustered data sets.

$\endgroup$
  • 1
    $\begingroup$ any thoughts to implementing this in R or adding partial dependency plot functionality $\endgroup$ – OliverFishCode Feb 5 '19 at 17:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.