I'm not even sure that the question makes much sense, but I think I saw a couple of titles of papers where they proposed random forest with random effects. Is this possible in R?
Currently, this paper (doi:10.1177/0962280220946080) does a revision of previous algorithms, including those cited in previous answers. Further, that paper introduce the R library
LongituRF, which allows to compute all those algorithms and the new ones.
Yeah it's possible. You should check out "RE-EM Trees: A Data Mining Approach for Longitudinal and Clustered Data," and the associated R package REEMtree.
It's been a while since I looked at the paper. I recall the authors had not yet tried forming ensembles of these trees, but that nothing suggested it wouldn't work.
Mixed Effects Random Forests (MERFs) are a thing. As the answer above states, there's some great research about them by Dr. Larocque's group at HEC Montreal. The paper is here: http://www.tandfonline.com/doi/abs/10.1080/00949655.2012.741599.
Essentially it is a theoretically sound way to combine the non-linear modeling of random forests with linear random effects.
We just released an open source package in Python implementing MERF using the above algorithm in the paper.
We wrote a detailed blog post about the package and how to use it for clustered data sets.
They are not commonly used together, and care should be taken before combining them.
Random forests are typically used as classifiers. The reason that you would use a random forest instead of another method (e.g. K-means clustering) is that you may have a large number of dimensions that you want to classify by. The issue with having the large number of dimensions is that if you wanted to test all combinations of dimension orders, you would have a large number of choices (it grows faster than the number of dimensions factorial).
Random effects are typically used in regression with repeated measures of the same thing. They are commonly used in mixed effects models where the term mixed refers to both fixed and random effects. The fixed effects are thought to represent the parameters that you will see again (e.g. a drug or a person's age). The random effects are thought to represent an instance of variability around a parameter that you will not see again (e.g. a specific person).
There are examples using them together when there is clustered data http://dx.doi.org/10.1080/00949655.2012.741599 and http://www2.ims.nus.edu.sg/Programs/014swclass/files/denis.pdf.
I'm unaware of any R packages that can do this analysis.
Instead of random forest, you can also use tree-boosting for the fixed effects part in a model with random effects. The GPBoost library with Python and R packages builds on LightGBM and allows for combining tree-boosting and mixed effects models. Simply speaking it is an extension of linear mixed effects models where the fixed-effects are learned using tree-boosting. See this blog post and Sigrist (2020) for further information.
Disclaimer: I am the author of the GPBoost library.