I have been looking through this overview of lm/lmer R formulas by @conjugateprior and got confused by the following entry:

Now assume A is random, but B is fixed and B is nested within A.

aov(Y ~ B + Error(A/B), data=d)

Below analogous mixed model formula lmer(Y ~ B + (1 | A:B), data=d) is provided for the same case.

I do not quite understand what it means. In an experiment where subjects are divided into several groups, we would have a random factor (subjects) nested within a fixed factor (groups). But how can a fixed factor be nested within a random factor? Something fixed nested within random subjects? Is it even possible? If it is not possible, do these R formulas make sense?

This overview is mentioned to be partially based on the personality-project's pages on doing ANOVA in R based itself on this tutorial on repeated measures in R. There the following example for the repeated measures ANOVA is given:

aov(Recall ~ Valence + Error(Subject/Valence), data.ex3)

Here subjects are presented with words of varying valence (factor with three levels) and their recall time is measured. Each subject is presented with words of all three valence levels. I do not see anything nested in this design (it appears crossed, as per the great answer here), and so I would naively think that Error(Subject) or (1 | Subject) should be appropriate random term in this case. The Subject/Valence "nesting" (?) is confusing.

Note that I do understand that Valence is a within-subject factor. But I think it is not a "nested" factor within subjects (because all subjects experience all three levels of Valence).

Update. I am exploring questions on CV about coding repeated measures ANOVA in R.

  • Here the following is used for fixed within-subject/repeated-measures A and random subject:

    summary(aov(Y ~ A + Error(subject/A), data = d))
    anova(lme(Y ~ A, random = ~1|subject, data = d))
  • Here for two fixed within-subject/repeated-measures effects A and B:

    summary(aov(Y ~ A*B + Error(subject/(A*B)), data=d))
    lmer(Y ~ A*B + (1|subject) + (1|A:subject) + (1|B:subject), data=d) 
  • Here for three within-subject effects A, B, and C:

    summary(aov(Y ~ A*B*C + Error(subject/(A*B*C)), data=d))
    lmer(Y ~ A*B*C + (1|subject) + (0+A|subject) + (0+B|subject) + (0+C|subject) + (0+A:B|subject) + (0+A:C|subject) + (0+B:C|subject), data = d)

My questions:

  1. Why Error(subject/A) and not Error(subject)?
  2. Is it (1|subject) or (1|subject)+(1|A:subject) or simply (1|A:subject)?
  3. Is it (1|subject) + (1|A:subject) or (1|subject) + (0+A|subject), and why not simply (A|subject)?

By now I have seen some threads that claim that some of these things are equivalent (e.g., the first: a claim that they are the same but an opposite claim on SO; the third: kind of a claim that they are the same). Are they?

  • 2
    $\begingroup$ Just a quick comment to say that, speaking strictly conceptually, in my opinion it virtually never makes sense to have a fixed factor nested in a random factor. I have read at least one textbook author say as much as well (can't remember reference at the moment). That said, it's possible that some of the model specifications you wrote above work out to be statistically equivalent to models that make more sense... I'd have to think about it more and play around with it a bit. $\endgroup$ Commented Aug 28, 2016 at 17:16
  • 4
    $\begingroup$ Actually I guess it makes sense if you think about the way R interprets the A/B syntax: it simply expands this to A + A:B. So if we consider a random term like subject/condition, this is conceptually dubious because it seems to suggest that conditions are nested in subjects, when clearly it's the opposite, but the model that is actually fit is subject + subject:condition, which is a perfectly valid model with random subject effects and random subject X slopes. $\endgroup$ Commented Aug 28, 2016 at 17:28
  • $\begingroup$ @JakeWestfall Thanks, this is along the lines I think about it myself by now, but I would very much like somebody to explain it properly. Actually I am surprised that this turns out to be a non-trivial question; I would have expected you to be one of the people who would answer is straight away. But this is a relief, as originally I thought my confusion must be dumb. By the way, is there some standard reference on lm and aov formulas? If I want to have an authoritative source on what exactly aov does (is it a wrapper for lm?) and how the Error() terms work, where should I look? $\endgroup$
    – amoeba
    Commented Aug 28, 2016 at 20:28
  • 1
    $\begingroup$ @amoeba Yes, aov is a wrapper for lm in the sense that lm is used for the least squares fit, but aov does some additional work (notably, translating the Error term for lm). The authoritative source is the source code or possibly the reference given in help("aov"): Chambers et al (1992). But I don't have access to that reference, so I'd look into the source code. $\endgroup$
    – Roland
    Commented Aug 29, 2016 at 7:54
  • $\begingroup$ Note for myself: here is a good cheat sheet uni-kiel.de/psychologie/rexrepos/posts/anovaMixed.html. Another one: rpsychologist.com/r-guide-longitudinal-lme-lmer. Also, Bates: stat.wisc.edu/~bates/UseR2008/WorkshopD.pdf and stat.ethz.ch/pipermail/r-sig-mixed-models/2009q1/001736.html $\endgroup$
    – amoeba
    Commented Oct 12, 2016 at 22:41

2 Answers 2


In mixed models the treatment of factors as either fixed or random, particularly in conjunction with whether they are crossed, partially crossed or nested can lead to a lot of confusion. Also, there appears to be differences in terminology between what is meant by nesting in the anova/designed experiments world and mixed/multilevel models world.

I don't profess to know all the answers, and my answer won't be complete (and may produce further questions) but I will try to address some of the issues here:

Does it make sense for a fixed effect to be nested within a random one, or how to code repeated measures in R (aov and lmer)?

(the question title)

No, I don't believe this makes sense. When we are dealing with repeated measures, then usually whatever the thing is that the measures are repeated on will be random, let's just call it Subject, and in lme4 we will want to include Subject on the right side of one or more | in the random part of the formula. If we have other random effects, then these are either crossed, partially crossed or nested - and my answer to this question addresses that.

The issue with these anova-type designed experiments seems to be how to deal with factors that would normally be thought of as fixed, in a repeated measures situation, and the questions in the body of the OP speak to this:

Why Error(subject/A) and not Error(subject)?

I do not usually use aov() so I could be missing something but, for me the Error(subject/A) is very misleading in the case of the linked question. Error(subject) in fact leads to exactly the same results.

Is it (1|subject) or (1|subject)+(1|A:subject) or simply (1|A:subject)?

This relates to this question. In this case, all the following random effects formulations lead to exactly the same result:

(1|subject) + (1|A:subject)
(1|subject) + (1|A:subject) + (1|B:subject)

However, this is because the simulated dataset in the question has no variation within anything, it is just created with Y = rnorm(48). If we take a real dataset such as the cake dataset in the lme4, we find that this will not generally be the case. From the documentation, here is the experimental setup:

Data on the breakage angle of chocolate cakes made with three different recipes and baked at six different temperatures. This is a split-plot design with the recipes being whole-units and the different temperatures being applied to sub-units (within replicates). The experimental notes suggest that the replicate numbering represents temporal ordering.

A data frame with 270 observations on the following 5 variables.

replicate a factor with levels 1 to 15

recipe a factor with levels A, B and C

temperature an ordered factor with levels 175 < 185 < 195 < 205 < 215 < 225

temp numeric value of the baking temperature (degrees F).

angle a numeric vector giving the angle at which the cake broke.

So, we have repeated measures within replicate, and we are also interested in the fixed factors recipe and temperature (we can ignore temp since this is just a different coding of temperature), and we can visualise the situation using xtabs:

> xtabs(~recipe+replicate,data=cake)

recipe 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
     A 6 6 6 6 6 6 6 6 6  6  6  6  6  6  6
     B 6 6 6 6 6 6 6 6 6  6  6  6  6  6  6
     C 6 6 6 6 6 6 6 6 6  6  6  6  6  6  6

If recipe were a random effect we would say that these are crossed random effects. In no way does recipe A belong to replicate 1 or any other replicate.

> xtabs(~temp+replicate,data=cake)

temp  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
  175 3 3 3 3 3 3 3 3 3  3  3  3  3  3  3
  185 3 3 3 3 3 3 3 3 3  3  3  3  3  3  3
  195 3 3 3 3 3 3 3 3 3  3  3  3  3  3  3
  205 3 3 3 3 3 3 3 3 3  3  3  3  3  3  3
  215 3 3 3 3 3 3 3 3 3  3  3  3  3  3  3
  225 3 3 3 3 3 3 3 3 3  3  3  3  3  3  3

Similarly for temp.

So the first model we might fit is:

> lmm1 <-  lmer(angle ~ recipe * temperature + (1|replicate), cake, REML= FALSE)

This will treat each replicate as the only source of random variation (other than the residual of course). But there could be random differences between recipes. So we might be tempted to include recipe as another (crossed) random effect but that would be ill-advised because we have only 3 levels of recipe so we can't expect the model to estimate the variance components well. So instead we can use replicate:recipe as the grouping variable which will enable us to treat each combination of replicate and recipe as a separate grouping factor. So whereas with the above model we would have 15 random intercepts for the levels of replicate we will now have 45 random intercepts for each of the separate combinations:

lmm3 <-  lmer(angle ~ recipe * temperature + (1|replicate:recipe) , cake, REML= FALSE)

Note that we now have (very slightly) different results indicating that there is some random variability due to recipe, but not a great deal.

We could likewise do the same thing with temperature.

Now, going back to your question, you also ask

Why (1|subject) + (1|A:subject) and not (1|subject) + (0+A|subject) or even simply (A|subject)?

I'm not entire sure where this (using random slopes) comes from - it doesn't seem to arise in the 2 linked questions - but my problem with (1|subject) + (1|A:subject) is that this is exactly the same as (1|subject/A) which means that A is nested within subject, which in turn means (to me) that each level of A occurs in 1 and only 1 level of subject which clearly is not the case here.

I will probably add to and/or edit this answer after I've thought about it some more, but I wanted to get my initial thoughts down.

  • $\begingroup$ Thanks a lot (+1). I am not sure I understand the cake dataset. It seems that replication is nested in recipe; the reason xtabs does not show it is exactly the reason you describe in your nested-vs-crossed answer: replication is confusingly coded as 1-15 and not as 1-45. For each recipe, 15 "replications" were made with 6 cakes; each cake was then baked at different temperature. So recipe is a between-subject factor and temperature is a within-subject factor. So according to that answer of yours, it should be (1|recipe/replicate). No? (1|replicate:recipe) is probably equivalent. $\endgroup$
    – amoeba
    Commented Aug 29, 2016 at 21:57
  • $\begingroup$ I focused my question only on within-subject factors, so it would be like restricting cake to only a single recipe. Regarding the third point that you say you are not sure where it comes from, please see the very last link in my Q, with an example of three within-subject factors. See also Jake's upvoted comment under this Q, where he mentions random slopes. $\endgroup$
    – amoeba
    Commented Aug 29, 2016 at 22:09
  • $\begingroup$ And regarding aov you are right that it seems that Error(subject/A) and Error(subject) yield the same results if there are no other factors, but take an example from the linked thread with two factors, and there Error(subject/(A*B)) and Error(subject) are not equivalent. My current understanding is that it is because the former includes random slopes. $\endgroup$
    – amoeba
    Commented Aug 30, 2016 at 0:23
  • $\begingroup$ @amoeba the cake dataset was not a good working example. My apologies. I will look a bit deeper into it and probably try to find a better one for illustration. $\endgroup$ Commented Aug 31, 2016 at 20:19
  • $\begingroup$ Thanks. Looking forward for any updates, as well as for the update that Placidia is preparing. In the meantime, I think I will a bounty here. $\endgroup$
    – amoeba
    Commented Aug 31, 2016 at 20:22

Ooooops. Alert commenters have spotted that my post was full of nonsense. I was confusing nested designs and repeated measures designs.

This site gives a useful breakdown of the difference between nested and repeated measures designs. Interestingly, the author shows expected mean squares for fixed within fixed, random within fixed and random within random -- but not fixed within random. It's hard to imagine what that would mean - if the factors in level A are chosen at random, then randomness now governs the selection of the factors of level B. If 5 schools are chosen at random from a school board, and then 3 teachers are chosen from each school (teachers nested in schools), the levels of the "teacher" factor are now a random selection of teachers from the school board by virtue of the random selection of schools. I can't "fix" the teachers I will have in the experiment.

  • 3
    $\begingroup$ +1, thanks a lot. Everything in your answer makes sense to me. However, I think we should agree that the word "nested" is being used in two distinct senses and this causes confusion. @RobertLong says that A is nested in B when each level of B occurs together with different levels of A. E.g. classes are nested within schools that are nested within towns etc. In your example, subjects are nested within the treatment/control factor. You say that time is nested within subjects, but all levels of time occur with all subjects so Robert would say they are crossed! This is a different "nested". Right? $\endgroup$
    – amoeba
    Commented Aug 30, 2016 at 9:08
  • 1
    $\begingroup$ In your example it is clear that time and treatment are fixed effects, but it is far from clear that time is nested within patients. Please can you provide a definition of "nested" ? $\endgroup$
    – Joe King
    Commented Aug 30, 2016 at 10:44
  • $\begingroup$ My mistake. I was confusing nesting and repeated measures. I have changed my answer -- again!. $\endgroup$
    – Placidia
    Commented Aug 30, 2016 at 15:50
  • $\begingroup$ Actually I liked your original answer with corrections/additions from earlier today. There was a lot of useful information for my question because as you see I am actually interested in "repeated measures" here (and the question about "nesting" was just a terminological point). I would suggest that you keep the previous revision! $\endgroup$
    – amoeba
    Commented Aug 30, 2016 at 15:58
  • 2
    $\begingroup$ As I was writing it, I realized that random effects in repeated measures are nested, and I want to test how the math works out and the degrees of freedom. I will amplify my answer when I am sure that I have this nailed! $\endgroup$
    – Placidia
    Commented Aug 30, 2016 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.