1
$\begingroup$

I have two categorical independent variables (Category: Before/After and Treatment [A,B,C]) and a binary response.

My regression output in R is as follows:

                              Estimate Std. Error z value Pr(>|z|)    
(Intercept)                   -1.16220    0.14103  -8.241  < 2e-16 ***
CategoryAfter                  0.89254    0.24876   3.588 0.000333 ***
TreatmentB                    -0.03485    0.20577  -0.169 0.865513    
TreatmentC                     0.29998    0.30781   0.975 0.329771    
CategoryAfter:TreatmentB      -0.73438    0.37841  -1.941 0.052295 .  
CategoryAfter:TreatmentC      -0.47215    0.56504  -0.836 0.403382

The interaction terms are not statistically significant. But "After" for Treatment A is significantly different from "Before" for Treatment A.

My understanding is that, if the interaction terms are not significant, then the results can't be interpreted as an interaction effect. However, I'm struggling to see how this isn't an interaction: the effect of Before/After was apparent only in Treatment A. How should I interpret this?

Here is the interaction plot: enter image description here

$\endgroup$
7
  • 1
    $\begingroup$ I suspect that the sample size is so small that extremely large effects are needed in order to come out significant. This doesn't mean that what you see is real though, unfortunately it means that there is large uncertainty about what goes on here. $\endgroup$ May 21 at 12:30
  • $\begingroup$ Thanks. That's fine. I obviously just had the wrong idea of what an interaction actually was! Sample size was almost 1000. $\endgroup$
    – Elemen00
    May 21 at 12:42
  • $\begingroup$ "Sample size was almost 1000" - somewhat surprising... other than by a small sample size, such a result can be explained by variances being very large compared to the effect sizes. If this is not the case (one would need to look at the data directly), something may have gone wrong in your execution of the analysis. You may also want to check for outliers. $\endgroup$ May 21 at 12:51
  • $\begingroup$ What was your categorical encoding? I'm mentioning it because if comes your encoding was indicator (dummy), then, in the presence of interaction, your main effect for "before-after" factor will actually measure it only in level A and not across the 3 levels. Therefore, the effect will appear exaggerated. $\endgroup$
    – ttnphns
    May 21 at 13:36
  • $\begingroup$ Yes @ttnphns it was an indicator variable, which I thought was the correct way of handling it (and R's default?). What should I have done instead? $\endgroup$
    – Elemen00
    May 21 at 14:41

1 Answer 1

0
$\begingroup$

There appears to certainly be a potential interaction present, it just doesn't appear to be statistically significant (which means the effect could simply be random noise). The key here is that the confidence intervals for the orange bars overlap. If they were not crossing, then one could say they are statistically significant.

This doesn't mean that the category factor is not meaningful or should be ignored. You can still report the descriptive differences as is, and perhaps highlight that further replication studies may illustrate if these effects are purely due to chance or not.

$\endgroup$
2
  • $\begingroup$ Thank you. I obviously thought I understood interactions better than I did! I figured that the ~0.05 p value had come from the only slight overlap between Treatment A/After and Treatment B/After, but had managed to get it in my head that a significant difference within a category was an interaction in itself. $\endgroup$
    – Elemen00
    May 21 at 12:37
  • 1
    $\begingroup$ @ChristianHennig I felt my second paragraph softened the language on whether or not the effect was "real", but I have edited my answer to make it more clear that this could also be totally random. $\endgroup$ May 21 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.