# When is it appropriate to report interactions?

I have two categorical variables, A and B. Each categorical variable has three levels(0,1,2). There is a certain dependent variable P against which I make a plot and see that there is an interaction between A and B. In my next step, I make a model when I regress upon P with A*B (model_interaction). When I look at the summary of this model I see that certain interactions terms are significant. Here is my question, is this enough evidence to say that there are significant interactions?

Along with the interaction model, I also made a linear model with A and B regressed on P (model_linear). When I compared model_interaction and model_linear I found no statistical difference between the two and I also found that the AIC score for model_linear was lower. So, after I've seen all of this do I still say that I have found significant interactions?

Just to summarize:

model_linear: P ~ A + B

model_interaction: P ~ A * B

Evidence for interaction:

1) Plots showing clear interaction.

2) Model with the interaction terms have significant p-values

Evidence against it:

1) Interaction model not significantly different from linear model

2) Linear model has lower AIC score compared to the interaction model.

Do I say there are interactions or not?

• "Interaction model not significantly different from linear model" — According to what test? (Above, you said that you "found no statistical difference"; I assume that "statistical" is a typo for "significant".) Apr 6, 2017 at 16:17
• My bad. I used the anova function from the car package to compare the two models.
– boY
Apr 6, 2017 at 16:38
• Do you mean Anova or anova? R is case-sensitive. anova is in stats whereas Anova is in car. Apr 6, 2017 at 16:40
• Did you include the main effects variables A and B along with A*B in the "interaction model"? If not I think you made the wrong comparison. Compare the main effects model to one with main effects plus interaction. Apr 6, 2017 at 17:02
• So in summary, OP, you compared two models (model_linear and model_interaction) with three methods, namely AIC and two different significance tests, and not all the methods agreed. It's not surprising to me that AIC and any given significance-testing approach disagreed, since AIC and significance testing have entirely different theory behind them, but I'm not sure how the two significance tests that you used differ. I would suggest looking into exactly what null hypothesis is being tested in each of these two cases: are you sure it's the same? Apr 6, 2017 at 17:12

I think you made a programming mistake. When comparing two lms that are the same except for the inclusion of an interaction term, anova should give the same $p$-value as summary gives for the interaction term. For example:

> coef(summary(lm(area ~ peri + shape * perm, data = rock)))["shape:perm",]
Estimate   Std. Error      t value     Pr(>|t|)
-10.71598140   4.84042362  -2.21385198   0.03219246
> anova(lm(area ~ peri + shape + perm, data = rock), lm(area ~ peri + shape * perm, data = rock))
Analysis of Variance Table

Model 1: area ~ peri + shape + perm
Model 2: area ~ peri + shape * perm
Res.Df      RSS Df Sum of Sq      F  Pr(>F)
1     44 74326644
2     43 66721703  1   7604941 4.9011 0.03219 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


See how the number 0.03219 appears in both outputs?

anova, at least for the case of two lms, uses an $F$-test, not a $χ^2$ test, and it is indeed equivalent to the $t$-test used by summary.lm.

As for AIC, you said:

The fact that these two [models] are not significantly different does not tell us which model is better than the other. To check which is better I was using the AIC scores to compare.

It's true that a non-significant result is uninformative, but it doesn't make sense to follow that up with AIC. The only reason to use a $p$-value for model selection is if you believe a priori that the simpler model is better and you'll stick with it unless the more complex model provides "enough" of an increase in fit. So if you're using the significance-testing approach, and you don't get a significant result, you should use the simpler model; end of story. AIC is a different approach with different standards of how to choose a model. Use the significance test or AIC, but not both. Of these, AIC is more sophisticated and is likely to be better for any real-world purpose.

• Thanks for the update! I think we are using different anova functions. The header of mine is this: Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)  Also, I found why the discrepancy is there. The anova function is basically checking for all interaction coefficients as zero together and in the regression model, it checks individually.
– boY
Apr 7, 2017 at 17:43
• @boY "All interaction coefficients", you say? Ah, I now see (thanks @Roland) that your IVs are categorical and hence the model P ~ A * B creates several interaction terms, not just one. Perhaps this is also why anova is using a $χ^2$ test instead of an $F$-test. Apr 7, 2017 at 19:50
• Yep. Strange that I didn't come across this in any text book.
– boY
Apr 7, 2017 at 20:07