# Testing contrasts in multilevel models

I'm trying to think of a way to test certain hypotheses regarding an experiment I've run. It's a 2 x 3 x 4 within-subjects, repeated-measures design. I would like to fit a (cross-classified) multilevel model with, say, intercepts varying over subjects.

Let's label the independent variables as A (2) x B (3) x C (4). The independent variables are dummy coded, the dependent variable is reaction time to a certain stimulus.

The specific question I'm interested in is whether the difference in reaction times between C1 and (C2 + C3 + C4)/3 is larger in condition A1 than in condition A2 (both averaged over levels of B).

Now, I could calculate the difference scores between C1 and the average of the remaining levels, and then run a t-test with A as the independent variable. However, since there are supposed to be benefits to using multilevel models, I want to try to get an answer out of them.

So, I assumed I have to fit the following model (in lme4 notation): reactionTime ~ (1 | subId) + A + B + C + A:C. If I'm not mistaken, the interaction is necessary to allow the C coefficients to be different with regard to levels of A.

If I fit the model, I get the following coefficients:

• Intercept
• A2
• B2
• B3
• C2
• C3
• C4
• A2xC2
• A2xC3
• A2xC4.

I understand that each combination of coefficients represents one experimental condition. However, I'm unsure on how to test the contrast I've specified.

Can I use the obtained coefficients to calculate the difference between C1 and (C2 + C3 + C4)/3 for A1 and A2 separately, and then perform a standard paired samples t-test on those values? If so, what would be the degrees of freedom and how would I find the SE of the difference?

Is there a direct way to test something like this in a multilevel model? I am aware that testing such "main effects" might be dubious, but I'd like to know whether it is even possible.

1. Have you run a variance components model, that is a model with just your outcome and the nesting structure specified in lmer? This will allow you calculate the variance partition coefficient/intraclass correlation coefficient, which tells you the degree to which variation in your outcome is between subID as such: $${}(VPC/ICC) =\frac{\sigma^2(subID)}{\sigma^2(subID)+\sigma^2(resid)}$$ If the VPC/ICC is very small (say <.01 - there is no agreed-upon cutoff), then maybe subject is not a major factor in modeling your outcome in which case you could revert to what might be the more familiar ANOVA framework.
3. Barring #2, check out the multcomp() package as I linked in my comment. See also https://cran.r-project.org/web/packages/multcomp/index.html
• (1) The ICC is .3 in my dataset (if I've understood correctly, the model I was supposed to fit is reactionTime ~ (1 | subId)). (2) (a) You're right about the averages. (b) Each row represents data from a participant in one out of ~150 trials (per participant), which have various combinations of A, B and C. I've thought about restructuring the data, but have the feeling that I'm losing information that way, and that's what I'm trying to avoid by using a LMM. (3) Will definitely check it out. Also found a simulation-based approach in Gelman & Hill, so I'll try to compare. Dec 13, 2019 at 17:24