4
$\begingroup$

I am running a mixed factors ANOVA for a brain imaging study on language processing. The design includes four within-subject factors:

  1. complexity: simple/complex;
  2. agreement: correct/number violation/gender violation;
  3. hemisphere: left/right;
  4. region: anterior/posterior;

and one between-subject factor:

  1. proficiency: low/high proficiency.

Factors 3 and 4 (hemisphere, region) only matter if they interact with the other factors. This is because the effects of complexity and agreement are predicted to emerge in specific areas of the scalp.

The results reveal a significant 5-way interaction (complexity by agreement by hemisphere by region by group), and I find it almost impossible to figure out what is driving the interaction by just looking at the means.

Am I justified in looking at the two proficiency groups separately? Most similar studies do this, but I'm unclear as to how to do this. For example, once I decide to examine proficiency learners separately, I find a 3-way interaction between agreement, hemisphere, and region in the high-proficiency learners (meaning that there is a brain response for agreement violations, which is captured in the left anterior portion of the scalp), but there is no such interaction for low-proficiency learners (p = .13) Is it licit to report this difference even if the original omnibus ANOVA (with group as a factor) showed that there was no agreement by hemisphere by region by proficiency interaction?

In other words, once I decide to look at the two proficiency groups, can I run the analysis as if I had never compared the two groups or am I constrained by the original ANOVA as to what follow-ups I can do?

$\endgroup$
1
  • 3
    $\begingroup$ You can Google general advice on complex interactions (but it probably won't be of much use). It would be best if you could post or link to the table of condition means. Also, report the magnitude and size of the lower order interactions and main effects. My first approach would be to graph it. You can probably do it in 4 panel graphs with 3 levels within each graph. $\endgroup$
    – John
    Jul 19, 2013 at 0:55

1 Answer 1

8
$\begingroup$

Without the numbers I can just give the general advice, graph it. You have 2 x 3 x 2 x 2 x 2 so that's not that hard. Take one of your 2 level variables and calculate it's effects across everything else. That's what you'll plot. You can now plot 3x2 or 2x3 panel graphs. You'd have 4 panels with the remaining two factors crossing them, one divided horizontally, and the other vertically.

Stare at that for awhile.

Maybe several permutations. Change what's on the x-axis, the panels, line types, and effects through as many iterations as you need until you come to an understanding. If you've got some theoretical reasons for directions of causality take the highest level and make it panels and the lowest and make it your line types or x-axis.

Hopefully you'll come up with something. This could take a lot of work and a long time to see the nature of the relationship. But going through several versions of the graphs can often allow you to see what's going on.

(I recently had to do the same thing for 10 x 3 x 2 x 2 x 2. It took months looking at graphs off and on and letting them incubate before the nature of the 5 way came through.)

UPDATE:I just noticed your question on splitting by proficiency. What you described is an absolute no-no. An interaction in one proficiency group while none in another does not tell you that there are differences between those groups. An interaction with proficiency would. So leaving proficiency out of your interactions gives you much less information. Your interactions mean something and the meaning is in the data, not in more tests.

As I ask most of my students, if you're going to test everything afterwards what was the point of doing an ANOVA in the first place?

$\endgroup$
7
  • $\begingroup$ Thanks a lot, John. This helped a lot. In this type of research, you usually plot the brain responses for the conditions under investigation (in this case: grammatical, number violation, gender violation) on a scalp map that represents the two brain hemispheres and the anterior/posterior portion of the head. So that kind of map takes care of two of the factors (hemisphere, anteriority) and functions as an interaction plot. I did this for the two proficiency groups to figure out what was going on, and it helped me decide that follow-ups to conduct. $\endgroup$ Jul 23, 2013 at 16:03
  • $\begingroup$ I also looked at all of the interaction plots that SPSS spits out and, of course, it was consistent. I felt a little uncomfortable deciding what follow-ups to do based on the visual observation of the means, but then again, I do not call those comparisons "planned comparisons" and I still corrected for Type I error, so it should be okay. Thanks! $\endgroup$ Jul 23, 2013 at 16:04
  • 1
    $\begingroup$ Just make sure that you extract as much information out of the interaction itself as you can and don't make inferential errors on followups. When all of your levels are 2 there is almost never any need for followup tests because the interactions tell you that what you see is significant. And NEVER do follow up tests and use something like x1:x2 was significant while y1:y2 was not therefore...The interaction already told you they were different while a followup like that does not. $\endgroup$
    – John
    Jul 23, 2013 at 16:13
  • $\begingroup$ Hi again, John. The main variable of interest in my study is agreement, which has 3 levels (grammatical, number violation, gender violation). So, I am running post-hoc tests, because I need to know whether number differs from grammatical, whether gender differs from grammatical, and whether number differs from gender. Does that sound reasonable? Thanks! $\endgroup$ Jul 24, 2013 at 16:34
  • $\begingroup$ It's unlikely you need the post hoc tests. The tests will probably make things worse inferentially. What if the scores are 3, 4, 5? Minimally the 3 and 5 will be different so you know that already but it's possible neither will be different from 4. What do you conclude from all of your tests? The ANOVA conclusion is there is a main effect and you describe the data, done. I have yet to see a 3 level ANOVA main effect where one's inference is improved by post hoc tests. For 5 and greater they can certainly be handy though. $\endgroup$
    – John
    Jul 24, 2013 at 16:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.