I have 3 within-subject factors, namely offset (1px, ..., 5px), side (left, right) and color (red, green), which define the characteristics of the stimulus in a reaction time experiment. The DV is reaction time RT. The design is fully balanced.

I ran a repeated measures ANOVA in R, like this:

options(contrasts = c("contr.helmert", "contr.poly"))

simon.aov <- aov(median.RT ~ color*side*offset + Error(VP / (color*side*offset)), data=dfa)

The results revealed a significant main effect of the color, as well as a significant interaction color x side and a significant 3-way interaction color x side x offset.
My primary focus lies on the interactions. Specifically, I want to know on which of the 5 offsets (i.e. on which levels of the third factor) the 2-way interaction color x side reaches significance.

I am by no means familiar with post-hoc contrasts and multiple comparisons, but this question is the gist of the thesis that I'm working on. So my progress depends on an adequate test to examine this question.

I highly appreciate any help on which test to run, and how to do this most efficiently in R.


I'm sorry I didn't provide any plots earlier.

@John: Here is the plot you requested.

Plot 1 of 3-way anova results

However, I believe, that this following plot rather clarifies my question:

Plot 2 of 3-way anova results

It seems like there is no color x side interaction at the first 3 levels of offset, but this interaction emerges at offset 4 and 5. This is what the plot seems to imply, however I don't know how to prove it statistically.

  • $\begingroup$ It would be great if you could post two graphs each with offset on the x-axis, colour as separate lines, and each graph a side. That would help in assisting. Or perhaps you could post, a 3-dimensional table reflecting same. The actual pattern of numbers is useful in giving you advice on this. Also, if you could post your predictions, especially for offset, that would be very helpful. It's unlikely that what you're stating you want to know is really what you need to know. $\endgroup$
    – John
    Commented Jun 27, 2013 at 17:12
  • $\begingroup$ I've added the plots. However, I frankly don't know, what exactly you mean by predictions or how to retrieve them in R from the aovlist object. $\endgroup$
    – vincentqu
    Commented Jun 27, 2013 at 20:56
  • $\begingroup$ Your predictions don't come from R, they come from what you expected to find in the experiment. And the best graph is the one that tells the story you want to tell best. You don't know what the story is yet. $\endgroup$
    – John
    Commented Jun 27, 2013 at 22:26
  • $\begingroup$ I'm basically interested in the interaction between color and side, i.e. red stimuli are responded to faster on the left side, and green stimuli on the right side (due to response button location). The main question is, how much these stimuli have to displaced for this effect to occur. Statistically speaking: on which levels of the factor offset does this color x side interaction occur? The second plot suggests, there is no such interaction on the levels 1, 2, 3. And based on the plot I'd say the interaction is present at offset 4 and 5. But is it statistically significant? $\endgroup$
    – vincentqu
    Commented Jun 28, 2013 at 12:39
  • $\begingroup$ Of course, I could just run a 2-way anova with factors color and side on the trials with offset 4 and 5 separately. But this doesn't seem quite efficient to me (due to alpha-error accumulation, for example) $\endgroup$
    – vincentqu
    Commented Jun 28, 2013 at 12:41

3 Answers 3


I'm not sure what you plan to test but looking at your first graph it seems pretty clear. There's generally an effect of colour but on the left side and at larger offsets it disappears.

I'm guessing you wanted to test all of the effects of colour to see where they were significant and where they weren't. If you found they were all significant or all not it wouldn't give your more information than your interaction (and the interaction is independent of such findings). If you found some were and some weren't it doesn't test your interaction because the difference between significant and not significant is not itself significant.

I suppose you could try to explore something else about the difference in patterns but given that offset seems like a continuous variable that should have some clear function if it is doing anything I'm not sure establishing anything else different about the waviness of the lines on the two sides would be where you'd want to go.

UPDATE after viewing comments

The explanation for your interaction would be that the effect of colour and side are consistent until offsets 4 and 5 where the effect of colour only exists for one of the sides. That's just a recasting of what I said before in line with your hypotheses.

Keep in mind what you'd have to test post hoc or in planned contrasts to really demonstrate this. Finding an effect just at large offsets is useless because colour itself is interacting with side; therefore you have to show it interacts with offsets to show they're having an effect as well. That's what your ANOVA is telling you. It's already the planned contrast you want. Is there anything else there that could be making that interaction occur? Do you need to explain anything else?

If you do the ANOVA at 4 and 5 you likely won't get an interaction with offset, just one between colour and side, which will be substantially less evidence for what you want to say, not more.

Keep in mind, interactions mean something. Look at your data and figure out what they mean before considering further statistical tests. If they're relatively clear, as in this case, then you're done.


Responding to the last question: At each offset, compute the color x side interaction contrast score for each subject and do a t-test on the mean. Then all you have to worry about is adjusting for multiplicity. I think a stepwise Bonferroni will suffice, but others may think differently.

On second thought, there is something else to worry about. Finding that the two-way interaction is significant at one offset and not at another does not by itself justify concluding that the two offsets have different two-way interactions. For that, you need to test and reject the 2 x 2 x 2 three-way interaction involving those two offsets. That means 10 more tests. The fact that the overall 2 x 2 x 5 three-way interaction is significant means only that some contrast of the five two-way interactions is signifcant. The logic here is much the same as in a one-way design where the overall test is significant and you want to know which means differ from which other means.


Generally speaking when in the presence of significant non-additivity ("interaction"), main effects and lower-order interactions are of less important. I usually find that by the time we have teased out a three-factor interaction, most of what is going on in the lower-order effects is explained.

I (tentatively) concur with your analysis of the second set of plots. The same pattern is present in the first set, but it isn't so obvious. But, as you asked, how to show this? You can show this via single-degree of freedom contrasts. This is the sort of situation where over-parameterized models (like the effects model you fitted) are a real pain.

What I would do is the following:

  1. Refit the model as RT ~ 0 + Color*Side*Offset;
  2. On a separate sheet of paper, lay out the 20 cells in the model;
  3. Define the Color MainEffect Contrast, the Side Main Effect Contrast, and the Offset Main Effect Contrasts, using the Helmert sequence stepping up from 0 offset;
  4. Use those Main effect contrasts to derive the four degrees of freedom for the Color x Side x Offset interaction;
  5. Do a step-up sequence of F-tests to see where the non-additivity becomes significant.

No, actually what I would do is the same thing using SAS because I understand how SAS handles linear models better than I grok R.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.