2
$\begingroup$

This is in the context of marking an undergraduate sociology essay in which the student has not found the predicted significant interaction effect in a Two-Way ANOVA.

The diagram below is schematic and doesn't depict the actual data. Imagine that the interaction effect is non-significant. The student has said that because the effect of task is significant in the treatment group but non-significant in the control group, there's some evidence that there really is an interaction effect.

enter image description here

I sense there is something fishy about this but I am struggling to come up with a precise explanation of why.

$\endgroup$
5
  • 1
    $\begingroup$ Comment 1: What if the task effect had p-value 0.051 in the control group but 0.049 in the treatment group? Comment 2: One has to be careful not to interpret "interaction effect is not statistically significant" as "there is no interaction effect". $\endgroup$
    – mark999
    Nov 23, 2013 at 7:49
  • 3
    $\begingroup$ It's quite possible for the effect of task is significant in the treatment group but non-significant in the control group when there is zero interaction. (Consider, for example, the case where the control group is smaller.) You might get some value from this article. $\endgroup$
    – Glen_b
    Nov 23, 2013 at 8:18
  • $\begingroup$ Glen_b, why not expand that a bit and make it the answer? $\endgroup$
    – John
    Nov 23, 2013 at 8:35
  • $\begingroup$ @Glen_b is correct. Nevertheless, the student is on to something: If the effect sizes are different in the two groups, there is evidence of an interaction. $\endgroup$
    – Peter Flom
    Nov 23, 2013 at 11:27
  • 1
    $\begingroup$ John, by itself it's not really a full answer. The difference between significance and non-significance isn't what hints there's an interaction -- as @PeterFlom suggests, a difference in effect size does. The problem is that you can't tell that possibility from what might arguably be equal population effects in the presence of sampling error (which is what the hypothesis test shows you). $\endgroup$
    – Glen_b
    Nov 23, 2013 at 22:06

1 Answer 1

3
$\begingroup$

The student wrote

because the effect of task is significant in the treatment group but non-significant in the control group, there's some evidence that there really is an interaction effect.

By itself, this is not very good evidence of an interaction effect, for reasons others have identified:

  1. p in treatment group could be 0.049 and in control group 0.051, this would not be evidence of anything in particular.

  2. Similarly, the effect size could be identical and, thus, the interaction exactly 0, and yet the p could be significant in one and not in the other due to different group sizes.

What, then, would be evidence of an interaction? That depends on whether you mean an important interaction or a significant one. The two are not the same. The importance of the interaction can be seen in a plot like the one in the question (only with real data!) or by examining the effect size in each group separately, or by looking at a regression with an interaction term and seeing how big that interaction term is (not how small its p-value is). The significance of the interaction can be examined by the p-value of the interaction term in the equation.

In addition to the usual reasons for not using p-values to indicate things they don't indicate, I will note that interaction terms are notorious in that, if the main effects that go into the interaction are measured with error (as nearly everything is, but usually more so in subjects like sociology than subjects like chemistry) then the error on the interaction term is even larger.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.