Testing contrasts in multilevel models

Question

I'm trying to think of a way to test certain hypotheses regarding an experiment I've run. It's a 2 x 3 x 4 within-subjects, repeated-measures design. I would like to fit a (cross-classified) multilevel model with, say, intercepts varying over subjects.

Let's label the independent variables as A (2) x B (3) x C (4). The independent variables are dummy coded, the dependent variable is reaction time to a certain stimulus.

The specific question I'm interested in is whether the difference in reaction times between C1 and (C2 + C3 + C4)/3 is larger in condition A1 than in condition A2 (both averaged over levels of B).

Now, I could calculate the difference scores between C1 and the average of the remaining levels, and then run a t-test with A as the independent variable. However, since there are supposed to be benefits to using multilevel models, I want to try to get an answer out of them.

So, I assumed I have to fit the following model (in lme4 notation): reactionTime ~ (1 | subId) + A + B + C + A:C. If I'm not mistaken, the interaction is necessary to allow the C coefficients to be different with regard to levels of A.

If I fit the model, I get the following coefficients:

• Intercept
• A2
• B2
• B3
• C2
• C3
• C4
• A2xC2
• A2xC3
• A2xC4.

I understand that each combination of coefficients represents one experimental condition. However, I'm unsure on how to test the contrast I've specified.

Can I use the obtained coefficients to calculate the difference between C1 and (C2 + C3 + C4)/3 for A1 and A2 separately, and then perform a standard paired samples t-test on those values? If so, what would be the degrees of freedom and how would I find the SE of the difference?

Is there a direct way to test something like this in a multilevel model? I am aware that testing such "main effects" might be dubious, but I'd like to know whether it is even possible.

Answer 1

This is a lot to tackle. Here are some initial thoughts and questions:

1. Have you run a variance components model, that is a model with just your outcome and the nesting structure specified in lmer? This will allow you calculate the variance partition coefficient/intraclass correlation coefficient, which tells you the degree to which variation in your outcome is between subID as such: $${}(VPC/ICC) =\frac{\sigma^2(subID)}{\sigma^2(subID)+\sigma^2(resid)}$$ If the VPC/ICC is very small (say <.01 - there is no agreed-upon cutoff), then maybe subject is not a major factor in modeling your outcome in which case you could revert to what might be the more familiar ANOVA framework.

2. Is there a way you might reconstruct your data so that the C variable gets exactly what you want directly out of the regression model? You are interested in the contrast between the average reaction time of C2, C3, and C4 vs C1, right? I'm not sure how your data is set-up other than it's "long" with each subject having multiple rows. What do the rows correspond to? If they correspond to the 4 C conditions, each then either being A1 or A2, could you not set them up so that they correspond to a C1 row and then an average of C2-C4 for each of A and B? Then your model would be similar except whatever you call the C2-C4 average would go in as a predictor.

3. Barring #2, check out the multcomp() package as I linked in my comment. See also https://cran.r-project.org/web/packages/multcomp/index.html