3
$\begingroup$

I ran a three way 2x2x2 ANOVA (AxBxC) that turned out to be just shy off a significant three way interaction (p = 0.06). I ran the two way interactions for AxB for the 2 levels of C anyway.

As it turns out, I underestimated how non-existent any form of 2 way interaction would be for level 2 of factor C. (partial eta squared = .02). For the first level of factor C the partial eta squared is a decent 0.456

So I'm probably underpowered ever so slightly for the three way interaction.

Is it going to be a statistical sin to decompose the separate two way interactions and report on them despite the three way interaction not being significant?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

What you are doing is the three-way interaction version of saying a significant simple effect and non-significant simple effect equals a significant interaction -- which is false. However, there is that famous saying that God surely loves the p=.06 as much as the p=.05. Really, though, what it sounds like is your hypothesis is slightly off, or at least what you thought your hypothesis was is slightly off.

Even if you could get your three-way interaction significant, if I were a reviewer on the paper (or a thesis advisor, etc.), I would check to make sure the pattern of results fits your hypotheses. It sounds like you expected the AB interaction component at C1 to go one way (let's say positively) and the AB interaction component at C2 to go the other way (let's say negatively). Even if there is a significant 3-way interaction, if your hypotheses really predict 4 simple effects of A that form a three-way interaction (e.g. significantly positive at B1C1, negative at B1C2, negative at B2C1, and positive at B2C2), then just your three-way interaction doesn't fully support your hypotheses. You would need a significant 3-way interaction, 2 significant simple interaction components (AB at C1, AB at C2), and then 4 significant simple effects in the predicted direction (A at B1C1, etc.).

All this is really to say, do your results mean you need to retool your hypotheses, and then run a replication that is properly powered to find your new, better hypotheses? If your hypotheses predict a significant interaction at C1 and then an attenuated effect at C2, then your data already does qualitatively fit the hypotheses, and the question is whether p=.06 is OK to report in a paper in your field (also you should be checking the simple effect of A at B1C1 and B2C1 for the significant AB interaction at C1). If it is not, then replication is still needed, and be more conservative with your expected effect size.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I have two main hypotheses. 1) There will be a main effect for A. This serves as my proto-manipulation check, so if this turns out to not be the case I'm dead in the water anyway. 2) This is where is gets tricky. My hypothesis here revolves around the conjoint effect of AB at the two different levels of C. In plain terms, I'm was trying to establish if my experimental manipulation from level C1 to C2 produced a reduction in the magnitude of the AB interaction. Instead, it seems to have nullified it completely. $\endgroup$
    – Rooirokbokkie
    May 13, 2015 at 17:39
  • $\begingroup$ I think I just have to accept that I don't have enough data to answer the question with the conservative approach I took to normalizing the response times and I'm trying not to go on a fishing trip. Since I underestimated the variability in the participant responses who also happen to deviated from the findings, I fell just short. I have to take my medicine and not do it again. Thanks for your patience and help @le_andrew $\endgroup$
    – Rooirokbokkie
    May 13, 2015 at 17:55
  • 1
    $\begingroup$ Ok, so you are predicting an attenuating interaction. First, know that these are rather low power predictions and also subject to other criticisms (for instance scale effects). More to the point, I don't see what's wrong with finding your interaction "nullified" at C2 rather than merely reduced. To me that says it had a stronger effect than you anticipated and the problem lies in the interaction not being strong enough at C1. $\endgroup$
    – le_andrew
    May 14, 2015 at 19:02
1
$\begingroup$

If you go down the route of 'decomposing' your interactions you should probably also adjust your p-values for multiple comparisons. In which case your study might still be underpowered to detect significant effects

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.