1
$\begingroup$

I have come across something that has got me kind of confused, I hope someone can help me out.

I did a mixed ANOVA with one within subjects factor with two levels 'time' and one between subjects factor 'group'. It was a comparison between a positive group and a negative group on scores pretraining and posttraining.

The results showed a significant main effect but no significant interaction effect. I should have stopped there maybe, but I did separate dependent t-tests for each group. These indicated that the scores of one group had changed significantly but the scores of the other group had not.

So now I don't understand anymore. Wouldn't a significant interaction effect indicate that there was a change in scores and that this change was affected by group (that the decrease in scores was significantly different between the groups)? Since there was no significant interaction shouldn't I find similar changes in both groups (either both significant or both not)?

Well, I hope this turns out to be easier than it seems to me right now, merci d'avance.

S.

$\endgroup$
  • $\begingroup$ I may be wrong but I believe part of what user53239 is confused about (that has not been specifically addressed in this thread..) is, given that s/he has NO SIG. INTERACTION, but S/HE DOES HAVE A SIG. MAIN EFFECT(S), what statistic can s/he report THAT determines a difference between her/his experimental groups? For example, should s/he do a post hoc test? If so, which one is appropriate? Is there even a standard follow up statistic (short of just looking at the differences between the means) for a sig. Main effect that can quantitatively determine whether the difference in Variable A $\endgroup$ – Tom B Jun 9 '17 at 1:32
1
$\begingroup$

It is easier than it seems. Significance is not a cutoff - it's not that things are significant, so there's a difference, or there isn't. Significantly different does not mean "different" and "not significantly different" does not mean "not different".

Imagine a succession of values: B is higher, but not significantly higher than A, C is higher, but not significantly higher than B, etc up to K is higher, but not significantly higher than J. But K is significantly higher than A. That makes sense, as each difference is small - you have the same issue.

This is (part of) the message of the paper by Gelman and Stern, called The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant, which you can find here: http://www.stat.columbia.edu/~gelman/research/published/signif4.pdf .

$\endgroup$
  • $\begingroup$ Hi Jeremy, thanks for your comment and for the article. It's still a bit weird to me though. Does this mean that I should always do separate tests for all groups after the anova even if it indicated that there was no sign. difference or is this something that should have been pointed out to me by post hoc tests or something? I thought that the anova replaced the separate tests. Now it seems rather useless. Also I wouldn't know how to report this; there was no sign. interaction, but group A showed a sign. decrease, whereas there was no sign. change in scores in group B? Thanks again :) $\endgroup$ – user53239 Aug 2 '14 at 10:40
  • $\begingroup$ Or is it, although the scores in group A decreased significantly, there was no significant difference between group A and group B? I don't think I'm supposed to report the t-tests then, but I just really want to understand. $\endgroup$ – user53239 Aug 2 '14 at 17:06
1
$\begingroup$

You should not be doing any separate t-tests, even if you have an interaction. You are throwing away valuable information. You have 4 groups. Separate t-tests only use the data from 2 of those groups at a time. You are essentially throwing away the data from the other 2 groups, which can be used to estimate the within group variance. Instead you can do simple effects tests. I don't know what software you use to do analysis, but the easy way to do that in R is to change your coding for the predictors. Main effects are obtained when you have sum-to-zero contrasts (e.g. -1 and 1 for the 2 levels of factor A, and -1 and 1 for the 2 levels of factor B). If you want the simple effect of A at a particular level of B, you dummy code B with 0 as the level of interest (e.g. -1 and 1 for A, 0 for level 1 of B and 1 for level 2 of B). Then switch the dummy coding of factor B to get the effect of A at level 2 of B. The model will always be the same summary(lm(dv~A*B,data=dat)). You'll notice the interaction effect is the same, because the interaction effect does not depend on how the factors are coded. The Effect of B will also be the same, because it is the difference attributable to that factor when A is 0, which held constant (0 is the mean of A: (-1+1)/2=0, so its the average effect of B across the levels A, which is the same as the main effect). The only thing that will change is the effect for A.

Regarding your confusion, Jeremy gives a good way to think about why you may have significant simple effects when you don't have a significant interaction. Put shorter, imagine the difference between level 1 and 2 of factor A at level B1 is equal to 1. It is significant with p=.05 (exactly). The difference at B2 is.9999. It's p value will be larger (p<.06). Your confusion may lie in the Gelman paper Jeremy linked to. Interactions don't tell you whether you have significant simple effect. They tell you whether the difference in slopes of your two regression lines is significant (i.e. whether the difference between level 1 and 2 of factor A at B1 is different than the difference between level 1 and 2 of factor A at B2 -- whether there is difference in the differences... the verbiage gets a little confusing). Its more complicated when you have continuous interactions or more than 2 levels of a predictor, but let's leave it there for now. While in the example before there is a significant simple effect and an non-significant one, you would definitely not think that the two sets of effects are different. They could hardly be any more the same. That is what the interaction tells you. Conversely, it is also entirely possible to have a significant interaction and all the simple effects you care about are not significant. For instance, take the previous example but change the simple effect of A at B1 to -.99. Both simple effects are not significant at alpha=.05, but the interaction will likely be significant.

Finally, your question really seems to be, "How do I conduct factorial/mixed ANOVA analysis with interactions." Lots of books offer step-by-step guides. They are not identical, but most are similar. I suggest Keppel and Wickens "Design and Analysis." Regarding this stage of the analysis, essentially, if you have a significant interaction, do simple effects tests. If you do not, interpret the main effects. You have a (possibly) significant simple effect and a (possibly) non-significant simple effect. However, you cannot statistically conclude that the two are different from each other. Thus, you should not be interpreting them separately. I said possibly before, because you'll have more power in the simple effects tests compared to your separate t-tests. However, without the interaction, you're really left with main effects anyway.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.