How many random effects to specify in lmer? I ran a computer-based experiment in which there were two within-subject factors, A and B. So all participants got multiple trials in each A*B cell. There was also one between subject factor, C.
I'm trying to predict response time, so initially I did:
> lmer(rt ~ A*B*C + (1|subj)

but was told I should specify random effect interactions as well, e.g.:
> lmer(rt ~ A*B*C + (1|subj) + (1|A:subj) + (1|B:subj)

In that case, shouldn't I also specify the three-way interaction? e.g.:
> lmer(rt ~ A*B*C + (1|subj) + (1|A:subj) + (1|A:B:subj)

I understand the first model, but I'm not quite clear on the other two--though they all provide different results. Can someone clarify what these models do and which one is most appropriate?
 A: With respect to the question which model is the most appropriate, I am still learning this for myself, but I can recommend an excellent paper covering this topic: 
Barr, DJ, Levy R., Scheepers C., Tily HJ. (2013) Random effects structure for confirmatory hypothesis testing: Keep it maximal, Journal of Memory and Language 68 (2013) 255–278
As the title implies, the bottomline is that you should always try to include the maximum random effects structure, because only that guarantees that you do not inflate Type I error (it is also an excellent tutorial on what random intercepts and slopes do in the regression model).
However, sometimes complex random effects structures can lead to problems with model convergence. That is why some researchers propose a little less radical approach, and suggest that you should keep the maximal random effects structure that is justified by data (i.e. you test whether the random effects improve the model fit by using anova() and strip it to significant random effects only). Florian Jaeger covers this issue in his excellent blog:
http://hlplab.wordpress.com/2009/05/14/random-effect-structure/
http://hlplab.wordpress.com/2011/06/25/more-on-random-slopes/
A: When you are specifying random effects in an lme4::lmer model, the random
factors go on the left of the pipe and the non-independence grouping variables
go on the right, so the fully specified model in your question would very
likely be:
lmer(rt ~ A*B*C + (A*B|subj))

I took some time to explore the difference between a random effect on the left
of the pipe to a random effect on the right side of the pipe but it made a
better post on it's own than as an answer to your particular question.
RPubs doc
Gist code
The most noticeable difference between the following two models...
lmer(rt ~ A + (1|subj/A))
lmer(rt ~ A + (A|subj))

...is that the latter estimates a random correlation parameter between random
intercepts and random slopes. If you remove that random correlation
parameter...
lmer(rt ~ A + (1|subj/A))
lmer(rt ~ A + (1|subj) + (0+A|subj))

...the two models return the exact same fixed effects (parameter estimates
and associated errors), though I would guess that similarity depends on
the particular design of the study.
