I'm not even sure that the question makes much sense, but I think I saw a couple of titles of papers where they proposed random forest with random effects. Is this possible in R?
They are not commonly used together, and care should be taken before combining them.
Random forests are typically used as classifiers. The reason that you would use a random forest instead of another method (e.g. K-means clustering) is that you may have a large number of dimensions that you want to classify by. The issue with having the large number of dimensions is that if you wanted to test all combinations of dimension orders, you would have a large number of choices (it grows faster than the number of dimensions factorial).
Random effects are typically used in regression with repeated measures of the same thing. They are commonly used in mixed effects models where the term mixed refers to both fixed and random effects. The fixed effects are thought to represent the parameters that you will see again (e.g. a drug or a person's age). The random effects are thought to represent an instance of variability around a parameter that you will not see again (e.g. a specific person).
There are examples using them together when there is clustered data http://dx.doi.org/10.1080/00949655.2012.741599 and http://www2.ims.nus.edu.sg/Programs/014swclass/files/denis.pdf.
I'm unaware of any R packages that can do this analysis.
Yeah it's possible. You should check out "RE-EM Trees: A Data Mining Approach for Longitudinal and Clustered Data," and the associated R package REEMtree.
It's been a while since I looked at the paper. I recall the authors had not yet tried forming ensembles of these trees, but that nothing suggested it wouldn't work.
Mixed Effects Random Forests (MERFs) are a thing. As the answer above states, there's some great research about them by Dr. Larocque's group at HEC Montreal. The paper is here: http://www.tandfonline.com/doi/abs/10.1080/00949655.2012.741599.
Essentially it is a theoretically sound way to combine the non-linear modeling of random forests with linear random effects.
We just released an open source package in Python implementing MERF using the above algorithm in the paper.
We wrote a detailed blog post about the package and how to use it for clustered data sets.