GLMERTREE with reponse in [0, 1] and multilevel design

Question

I have multilevel data (with nested random effects: (1 | cluster-of-cluster/cluster) in lme4 syntax) where the response is a continuous variable between $[0, 1]$ (i.e., including 0 and 1).

Usually, I'd model this data using a generalized linear mixed-effects model (e.g., with glmmTMB: https://cran.r-project.org/web/packages/glmmTMB/index.html), possibly after transforming the response as in Smithson and Verkuilen, 2006 (the "better lemon squeezer" paper: https://doi.apa.org/doiLanding?doi=10.1037%2F1082-989X.11.1.54), where $y' = \frac{y * (N - 1) + (1/2)}{N}$.

I want to model the data using recursive partitioning. glmertree (https://cran.r-project.org/web/packages/glmertree/index.html) is the closest to what I need. I'd be using models similar to the one in section 2 of the glmertree vignette (https://cran.r-project.org/web/packages/glmertree/vignettes/glmertree.pdf), where the node-specific model is just an intercept, and in section 3, where a discrete-level treatment effect is fitted in the nodes.

My questions are:

1. Do I gain anything using Smithson and Verkuilen's transformation? I am not fitting a beta model.

2. I am using a gaussian linear-mixed effects model underneath. Are there better options?

   I thought about using ctree, from partykit (https://cran.r-project.org/web/packages/partykit/index.html), with the default identity function for influence, which I think makes no particular distributional assumptions for this case. But I have multilevel data, and different partitioning variables are measured at different levels (observation, clusters of observations, and clusters of clusters), and using the cluster argument in ctree precludes finding splits in variables measured at the cluster and 'cluster of cluster' levels (I've run some toy examples, and in p. 28 of the vignette, https://cran.r-project.org/web/packages/partykit/vignettes/ctree.pdf says: "the variance of the test statistics used for variable selection and also splitting is computed separately, leading to stratified permutation tests (in the sense that only observations within clusters are permuted)").

3. lmertree also accepts a cluster argument but I am not sure I really should use it. Toy examples I have simulated show that using it vs. not using it does not affect like in ctree and generally has minor effects but its usage is discussed in section 4 of the vignette and in Maaike Jorink's MSc's thesis (https://openaccess.leidenuniv.nl/bitstream/handle/1887/69342/Jorink%2C%20Maaike-s1907387-MA%20Thesis%20MS-2018.pdf).

Answer 1

Before addressing your three questions, I would like to point out that the answers depend on which model you really want to fit to the data.

Regressors: I think in terms of predictors you want to have (a) fixed effects that capture a treatment-subgroup interaction (where the appropriate groups have to be learned from the data) and (b) random effects capturing the correlation from the nested cluster structure.

Response: In terms of response distribution you say that your data are in $[0, 1]$ but that you don't want to fit a beta distribution. But do you adopt another probabilistic model that can fit such data well? Or are you adopting an approximately normal model and then end up using it as a conditional mean model rather than a conditional distributional model?

Trees with mixed effects: The general strategy implemented in glmertree that we slightly extended from the idea underlying REEMtree is to iterate between estimation of the groups for the fixed effects (via a tree, given a specification of the random effects) and estimation of the random effects (via a mixed-effects model, given the groups from the tree). In REEMtree this is done by estimating the random effects, subtracting them from the response, and then fitting the tree to the adjusted response - before doing it vice versa. In glmertree we are slightly more general by treating one of the effects as an offset and then estimating the other. The advantage is that you can then fit the resulting model as a single mixed-effects model and you converge a little bit faster. If you are planning to use an approximately normal response, then you can try out either REEMtree (based on ctree and nlme) or lmertree (based on lmtree and lme4). Or you can also try to "roll your own" algorithm by plugging a suitable tree flavor together with glmmTMB for example. The iterations between the two models typically converge very quickly.

Answer to your questions:

1. The ad-hoc transformation by Smithson & Verkuilen is just a "trick" to make the response fit in (0, 1) when it would actually by [0, 1]. This is only useful if you fit a model that needs the response to be in the open interval (such as a standard beta distribution) and if the extreme values 0 and 1 in the original scaling are driven by the same effects as observations slightly larger than 0 or smaller than 1, respectively. Thus: If you're not fitting a beta distribution I guess that you don't need the transformation - but I wonder how your model then fits the data (see above).
2. My feeling is that what cluster in ctree does is too strict for your data. It really treats the data as separate blocks, only permuting within blocks.
3. In contrast lmtree (which is underlying lmertree) just adjusts the inference for within-cluster correlation. The latter typically only has an effect if explanatory variables are correlated with the clustering variable. If these are approximately uncorrelated, however, the cluster argument has almost no effect. See also the simulation study in vignette("sandwich-CL", package = "sandwich").

I hope that this gets you a few steps further in your modeling approach!