I ran a 2 x 2 x 2 full factorial repeated measures experiment where 20 participants were exposed 30 times to all combinations of the factors A, B and C in random order. This is a standard procedure in response time experiments.

A1 - B1 - C1 
A2 - B1 - C1 
A1 - B2 - C1 
A2 - B2 - C1 
A1 - B1 - C2
A2 - B1 - C2
A1 - B2 - C2
A2 - B2 - C2

I'm having a hard time figuring out how to optimally analyse the data using a mixed effects model. I've specified the factors as fixed and the participants with random intercepts, but I am not sure whether this is the correct thing to do.

My main hypothesis was that there will be a significant interaction between factors A and B, but only at level 1 of factor C. I'm interested specifically in the interaction of A and B at the levels of C since I predicted that the combination of (A1B1) - (A1B2) will be significantly higher than (A2B1) - (A2B2), but only for C1.

enter image description here

When I ran the model to see whether there was a three-way interaction between A, B, and C, I found that the interaction was not statistically significant.

(interaction_full<-lmer(dv ~ A * B * C + (1|participant), data_full))

When I ran separate models for the levels of C it turns out that I was right. There is a major A X B interaction for C1, but not for C2. Would this be problematic?

(interaction_C1<-lmer(dv ~ A * B + (1|participant), data_C1))

(interaction_C2<-lmer(dv ~ A * B + (1|participant), data_C2))

enter image description here

Where do I go from here to address my hypothesis? Would it be sufficient to run separate models for C1 and C2 and then indicate that because there was a significant A X B interaction when running the models on C1, but not on C2 that I can reject the null?

p.s Load is A, compatibility is B and salience is C.

  • $\begingroup$ What do you mean by "Would this be problematic?" What do you foresee would be problematic and why? $\endgroup$ Commented May 10, 2015 at 3:55
  • $\begingroup$ What is your hypothesis using your figure terms? Is it that incompatible and neutral are different at low load (also salient vs non-salient)? Or is it that incompatible and neutral have different change from high load to low load (with effect of salient vs non-salient)? Or is it just that incompatible are different from neutral? $\endgroup$
    – rnso
    Commented May 10, 2015 at 9:52
  • $\begingroup$ The basic hypothesis is that incompatible and neutral are different but only for low load; not for high load. This is know as the compatibility effect in cognitive science. My main hypothesis though is that this will be nullified by the introduction of salient distractors but will still be valid for non-salient distractors. This appears to be the case, but have I done enough to actually demonstrate this? $\endgroup$ Commented May 10, 2015 at 10:21

1 Answer 1


As I understand it, this issue is not at all specific to mixed models, but is a variant of "the difference between significant and non-significant is not significant" (see also other blog posts by Gelman); see also Pockock et al.2002. Consider the following statements:

  • "within C1, we can reject the null hypothesis of no $A \times B$ interaction; the effect of A on responses differs significantly across levels of B (and vice versa)"
  • "within C2, we cannot reject the null hypothesis; the effect of A on responses does not differ significantly ..."
  • "we cannot reject the hypothesis that the $A \times B$ interaction is the same in both levels of C"

All three are true. As @Wolfgang points out below, the test of the three-way interaction is the formal test that there is a difference between the strength of the interaction across different levels.

The corresponding picture:

enter image description here

  • $\begingroup$ Thanks for the comments and article Ben. I guess I'm just unsure whether or not this informal comparison of the two models for the two levels of C would be considered sufficient to reject the null. $\endgroup$ Commented May 10, 2015 at 8:36
  • 2
    $\begingroup$ No, that's in fact the point of the article by Gelman -- a point that has been made others before (e.g. Pocock et al., 2002). Suppose the p-values for the 2-way interaction are .04 and .06 within the two levels of C. Would you conclude that the interaction is really different for the two levels, that is, that there is a 3-way interaction? I hope not. Now your p-values may be more dissimilar, but how different would they have to be to decide that there really is a 3-way interaction? Well, that's what the test for the 3-way interaction is for! $\endgroup$
    – Wolfgang
    Commented May 10, 2015 at 10:37
  • $\begingroup$ Ah. That actually makes sense. What type follow-up analyses would I have to look at to break down the relationship if there was a three way interaction? When doing normal ANOVA's the next step would be to break down the separate two way interactions for factor C the way I did if there was a 3 way interaction. Would this still be acceptable in this particular instance? $\endgroup$ Commented May 10, 2015 at 11:23
  • 2
    $\begingroup$ Yes, if there really is a 3-way interaction, then you would examine how the 2-way interactions differ across the levels of the third factor. Again, this is nothing different than what one would do in an ANOVA, a mixed-effects model, logistic regression, or pretty much any other modeling framework. $\endgroup$
    – Wolfgang
    Commented May 10, 2015 at 18:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.