Using participant-specific slopes in correlations and dependent variables in subsequent models

Question

I have been asked to review a paper where the authors have intensive longitudinal data (several observations per day for several days). So they have lots of observations per participant. They do three things that strike me as wrong:

1. they run a multilevel regression and save the participant-specific random slope coefficients. Then, they compute within-person correlations between those participant-specific slope coefficients and the (raw) momentary variable that was the dependent variable in the first multilevel regression and present these correlations as one of the findings.

2. they run several multilevel regressions, save the participant-specific random slopes and correlate them with each other (within-person), and present these correlations as one of the findings.

3. from the multilevel regression mentioned in #1, they take the participant-specific random slope coefficients and then use them in a subsequent analysis as the dependent variable (or possible as mediator, it's a bit unclear).

All this seems wrong to me, but I'm not sure my feeling is correct, and if so, why. I have a vague idea that these procedures would make standard errors artificially small (in the latter analyses) but that's about it. Also the authors use very sophisticated analyses otherwise so they seem to know what they're doing (I know it's no guarantee but I feel a bit uncomfortable criticizing them).

Is my intuition completely wrong or is the above questionable?

Answer 1

This is a good question and the concerns you have are justified. I will address the second and third questions you asked. I honestly have no idea what they were trying to accomplish in the first analysis you mentioned.

they run several multilevel regressions, save the participant-specific random slopes and correlate them with each other (within-person), and present these correlations as one of the findings.

The problem here is the one I mentioned in my comment, that the random effects have a lot of uncertainty associated with them. To get a sense of this, one can use the ranef() function on a g/lmer() model to see the predictions.

library(lme4)
data(sleepstudy)
m1 <- lmer(Reaction ~ Days + (Days|Subject), sleepstudy)
rr <- ranef(m1)

Here they are, below. Note that because all subjects were measured the same number of occasions, the standard errors are the same. If a subject had fewer or more occasions, their SE would be different.

> head(as.data.frame(rr),2)
   grpvar        term grp    condval   condsd
1 Subject (Intercept) 308   2.258551 12.07086
2 Subject (Intercept) 309 -40.398738 12.07086
> tail(as.data.frame(rr),2)
    grpvar term grp    condval   condsd
35 Subject Days 371 -0.9881562 2.304839
36 Subject Days 372  1.2840221 2.304839

So, when you assign a single value (condval) to an individual, you are saying that you are 100% certain that this is their intercept or slope value, when in reality, the model is much less certain of that.

What can you do instead? You can run a multivariate random effects model. This ensures that all random effects preserve their uncertainty and as part of the model output, you get correlations (or covariances) between all random effects. I would probably use brms() for this in R. See here.

from the multilevel regression mentioned in #1, they take the participant-specific random slope coefficients and then use them in a subsequent analysis as the dependent variable (or possible as mediator, it's a bit unclear).

If indeed they are taking the predicted slope values and using them as outcomes in other models, then this is completely unnecessary. The mixed effect model allows one to model slopes as outcomes. See here. It's accomplished by specifying a random slope, and then interacting any cluster-level predictors (say the model is students in schools) with the variable that has been specified as varying across the clusters.

If instead they did some kind of mediation, then I would need to know more about it. It is possible that such could be done within a multilevel structural equation model, but as before, you do it all within a single model so that the uncertainty in the random effects is accounted for properly. See example F in this pdf from Kristopher Preacher. There, he is showing syntax for Mplus, which is a powerful stand-alone structural equation modeling program.