2x2x5 repeated measures ANOVA: significant 3-way interaction

Question

I have 3 within-subject factors, namely offset (1px, ..., 5px), side (left, right) and color (red, green), which define the characteristics of the stimulus in a reaction time experiment. The DV is reaction time RT. The design is fully balanced.

I ran a repeated measures ANOVA in R, like this:

options(contrasts = c("contr.helmert", "contr.poly"))

simon.aov <- aov(median.RT ~ color*side*offset + Error(VP / (color*side*offset)), data=dfa)

The results revealed a significant main effect of the color, as well as a significant interaction color x side and a significant 3-way interaction color x side x offset.
My primary focus lies on the interactions. Specifically, I want to know on which of the 5 offsets (i.e. on which levels of the third factor) the 2-way interaction color x side reaches significance.

I am by no means familiar with post-hoc contrasts and multiple comparisons, but this question is the gist of the thesis that I'm working on. So my progress depends on an adequate test to examine this question.

I highly appreciate any help on which test to run, and how to do this most efficiently in R.

Edit:

I'm sorry I didn't provide any plots earlier.

@John: Here is the plot you requested.

However, I believe, that this following plot rather clarifies my question:

It seems like there is no color x side interaction at the first 3 levels of offset, but this interaction emerges at offset 4 and 5. This is what the plot seems to imply, however I don't know how to prove it statistically.

Answer 1

I'm not sure what you plan to test but looking at your first graph it seems pretty clear. There's generally an effect of colour but on the left side and at larger offsets it disappears.

I'm guessing you wanted to test all of the effects of colour to see where they were significant and where they weren't. If you found they were all significant or all not it wouldn't give your more information than your interaction (and the interaction is independent of such findings). If you found some were and some weren't it doesn't test your interaction because the difference between significant and not significant is not itself significant.

I suppose you could try to explore something else about the difference in patterns but given that offset seems like a continuous variable that should have some clear function if it is doing anything I'm not sure establishing anything else different about the waviness of the lines on the two sides would be where you'd want to go.

UPDATE after viewing comments

The explanation for your interaction would be that the effect of colour and side are consistent until offsets 4 and 5 where the effect of colour only exists for one of the sides. That's just a recasting of what I said before in line with your hypotheses.

Keep in mind what you'd have to test post hoc or in planned contrasts to really demonstrate this. Finding an effect just at large offsets is useless because colour itself is interacting with side; therefore you have to show it interacts with offsets to show they're having an effect as well. That's what your ANOVA is telling you. It's already the planned contrast you want. Is there anything else there that could be making that interaction occur? Do you need to explain anything else?

If you do the ANOVA at 4 and 5 you likely won't get an interaction with offset, just one between colour and side, which will be substantially less evidence for what you want to say, not more.

Keep in mind, interactions mean something. Look at your data and figure out what they mean before considering further statistical tests. If they're relatively clear, as in this case, then you're done.

Answer 2

Responding to the last question: At each offset, compute the color x side interaction contrast score for each subject and do a t-test on the mean. Then all you have to worry about is adjusting for multiplicity. I think a stepwise Bonferroni will suffice, but others may think differently.

On second thought, there is something else to worry about. Finding that the two-way interaction is significant at one offset and not at another does not by itself justify concluding that the two offsets have different two-way interactions. For that, you need to test and reject the 2 x 2 x 2 three-way interaction involving those two offsets. That means 10 more tests. The fact that the overall 2 x 2 x 5 three-way interaction is significant means only that some contrast of the five two-way interactions is signifcant. The logic here is much the same as in a one-way design where the overall test is significant and you want to know which means differ from which other means.

Answer 3

Generally speaking when in the presence of significant non-additivity ("interaction"), main effects and lower-order interactions are of less important. I usually find that by the time we have teased out a three-factor interaction, most of what is going on in the lower-order effects is explained.

I (tentatively) concur with your analysis of the second set of plots. The same pattern is present in the first set, but it isn't so obvious. But, as you asked, how to show this? You can show this via single-degree of freedom contrasts. This is the sort of situation where over-parameterized models (like the effects model you fitted) are a real pain.

What I would do is the following:

1. Refit the model as RT ~ 0 + Color*Side*Offset;
2. On a separate sheet of paper, lay out the 20 cells in the model;
3. Define the Color MainEffect Contrast, the Side Main Effect Contrast, and the Offset Main Effect Contrasts, using the Helmert sequence stepping up from 0 offset;
4. Use those Main effect contrasts to derive the four degrees of freedom for the Color x Side x Offset interaction;
5. Do a step-up sequence of F-tests to see where the non-additivity becomes significant.

No, actually what I would do is the same thing using SAS because I understand how SAS handles linear models better than I grok R.