Let's say we have two models specified by the following formulas in R's lmer():

i) Y ~ A + B + (A:B|SUBJECT)

ii) Y ~ A + B + (SUBJECT|A:B)

For the random effects, equation i) specifies a random slope and intercept for each level of A:B by subject, as equation ii) specifies a random slope and intercept for each subject by level of A:B (if i am not interpreting this wrongly).

For both these model formulas, what are the differences in the linear mixed effects model equation regarding the random effects? And is the model ii) wrongly specified?

  • $\begingroup$ The grouping variable specifying the repeated measures belongs on the RHS of |. The variable name SUBJECT usually indicates that this specifies the experimental units which have been investigated repeatedly. Thus model 1 appears to be the correct one. $\endgroup$
    – Roland
    Jul 1, 2016 at 11:04
  • $\begingroup$ Thanks for the answer, i had thought that model 1 was the correct one. However i am getting a slope on the residuals plot for this model, which seems to be violating an assumption of the model, any idea how to circumvent this? $\endgroup$ Jul 1, 2016 at 11:46
  • $\begingroup$ You need to give way more information to answer this. You don't even say what Y actually is, maybe a gaussian model is not appropriate for the residuals. There also might be problems of influential data points, heterogeneity or even autocorrelation. $\endgroup$
    – Roland
    Jul 1, 2016 at 11:50
  • $\begingroup$ The Y is a continuous variable (MIC) representing the antibiotics concentration needed to kill a microorganism, which was transformed to log10 scale. Is there any literature you can point that covers those diagnostics? $\endgroup$ Jul 1, 2016 at 11:53
  • $\begingroup$ Why was it transformed to the log10 scale? Are you sure that an exponential relationship with a multiplicative error model is appropriate? (As a starting point I usually recommend this book.) $\endgroup$
    – Roland
    Jul 1, 2016 at 12:03

1 Answer 1


In model i) the right side of | specifies that observations are nested/grouped within SUBJECT, resulting in the estimation of a random intercept (that is, for each level of SUBJECT, that level's intercept's deviation from the fixed intercept). The left side of the | specifies that the interaction between A and B varies for each level of SUBJECT, that is, a random slope for the interaction. Normally this would be the deviation from the fixed estimate of the interaction. However the fixed part of your model formula does not contain the interaction, so in this case it is the deviation from zero.

In model ii) you have the interaction between A and B as the grouping variable, resulting in a random intercept for the interaction, and a random slope for SUBJECT so that the effect of SUBJECT varies for each level of the interaction.

Both models will also estimate a correlation between the random slope deviations and random intercept deviations.

Note that model i) is a little strange because you don't have the interaction as a fixed effect. A more common situation would be to have one or both both main effects as fixed effects and random effects, and the interaction as a fixed effect, both, or not at all.

Model ii) is rather more strange because it seems unlikely that the interaction could be considered as a grouping variable.

  • $\begingroup$ So model 1 should be something like Y ~ A:B + (A:B|SUBJECT), Y ~ A*B + (A:B|SUBJECT), or Y ~ A + B + (A|SUBJECT) + (B|SUBJECT) ? $\endgroup$ Jul 1, 2016 at 12:50
  • $\begingroup$ It depends on what the physical/biological theory suggests. I might start with Y ~ A*B + (1|SUBJECT) and assess how that fits the data before adding random slopes and assessing if more complex models fit any better. From there, a logical progression is Y ~ A*B + (A+B|SUBJECT) and then Y~A*B+ (A*B|SUBJECT) but note that the increasing complexity results in many more parameters to be estimated (the latter model will estimate 4 random effects and 6 correlations) so you might want to fit models with uncorrelated random effects eg Y ~ A*B + (A|SUBJECT) + (0+B|SUBJECT) + (0+A:B|SUBJECT) $\endgroup$ Jul 1, 2016 at 13:16
  • $\begingroup$ Lets say that A is a between subjects factor and B a within subjects factor, where the DV was measured for every level of B, and each subject falls in one of the two levels of A. Note that B is not repeated $\endgroup$ Jul 1, 2016 at 14:55
  • $\begingroup$ Please could you write a new question about how to model your data. Ideally provide an str(data) as well as the information about how each factor changes. Let this question and answer stay about the model formula meaning. $\endgroup$ Jul 1, 2016 at 15:13
  • $\begingroup$ Done, stats.stackexchange.com/questions/221721/… $\endgroup$ Jul 1, 2016 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.