2
$\begingroup$

I have two categorical variables, A and B. Each categorical variable has three levels(0,1,2). There is a certain dependent variable P against which I make a plot and see that there is an interaction between A and B. In my next step, I make a model when I regress upon P with A*B (model_interaction). When I look at the summary of this model I see that certain interactions terms are significant. Here is my question, is this enough evidence to say that there are significant interactions?

Why am I asking this?

Along with the interaction model, I also made a linear model with A and B regressed on P (model_linear). When I compared model_interaction and model_linear I found no statistical difference between the two and I also found that the AIC score for model_linear was lower. So, after I've seen all of this do I still say that I have found significant interactions?

Just to summarize:

model_linear: P ~ A + B

model_interaction: P ~ A * B

Evidence for interaction:

1) Plots showing clear interaction.

2) Model with the interaction terms have significant p-values

Evidence against it:

1) Interaction model not significantly different from linear model

2) Linear model has lower AIC score compared to the interaction model.

Do I say there are interactions or not?

Improve this question
$\endgroup$
12
  • 1
    $\begingroup$ "Interaction model not significantly different from linear model" — According to what test? (Above, you said that you "found no statistical difference"; I assume that "statistical" is a typo for "significant".) $\endgroup$
    – Kodiologist
    Commented Apr 6, 2017 at 16:17
  • $\begingroup$ My bad. I used the anova function from the car package to compare the two models. $\endgroup$
    – boY
    Commented Apr 6, 2017 at 16:38
  • 2
    $\begingroup$ Do you mean Anova or anova? R is case-sensitive. anova is in stats whereas Anova is in car. $\endgroup$
    – Kodiologist
    Commented Apr 6, 2017 at 16:40
  • 1
    $\begingroup$ Did you include the main effects variables A and B along with A*B in the "interaction model"? If not I think you made the wrong comparison. Compare the main effects model to one with main effects plus interaction. $\endgroup$
    – Michael R. Chernick
    Commented Apr 6, 2017 at 17:02
  • 1
    $\begingroup$ So in summary, OP, you compared two models (model_linear and model_interaction) with three methods, namely AIC and two different significance tests, and not all the methods agreed. It's not surprising to me that AIC and any given significance-testing approach disagreed, since AIC and significance testing have entirely different theory behind them, but I'm not sure how the two significance tests that you used differ. I would suggest looking into exactly what null hypothesis is being tested in each of these two cases: are you sure it's the same? $\endgroup$
    – Kodiologist
    Commented Apr 6, 2017 at 17:12

1 Answer 1

Reset to default
2
$\begingroup$

I think you made a programming mistake. When comparing two lms that are the same except for the inclusion of an interaction term, anova should give the same $p$-value as summary gives for the interaction term. For example:

> coef(summary(lm(area ~ peri + shape * perm, data = rock)))["shape:perm",]
    Estimate   Std. Error      t value     Pr(>|t|) 
-10.71598140   4.84042362  -2.21385198   0.03219246 
> anova(lm(area ~ peri + shape + perm, data = rock), lm(area ~ peri + shape * perm, data = rock))
Analysis of Variance Table

Model 1: area ~ peri + shape + perm
Model 2: area ~ peri + shape * perm
  Res.Df      RSS Df Sum of Sq      F  Pr(>F)  
1     44 74326644                              
2     43 66721703  1   7604941 4.9011 0.03219 *
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

See how the number 0.03219 appears in both outputs?

anova, at least for the case of two lms, uses an $F$-test, not a $χ^2$ test, and it is indeed equivalent to the $t$-test used by summary.lm.

As for AIC, you said:

The fact that these two [models] are not significantly different does not tell us which model is better than the other. To check which is better I was using the AIC scores to compare.

It's true that a non-significant result is uninformative, but it doesn't make sense to follow that up with AIC. The only reason to use a $p$-value for model selection is if you believe a priori that the simpler model is better and you'll stick with it unless the more complex model provides "enough" of an increase in fit. So if you're using the significance-testing approach, and you don't get a significant result, you should use the simpler model; end of story. AIC is a different approach with different standards of how to choose a model. Use the significance test or AIC, but not both. Of these, AIC is more sophisticated and is likely to be better for any real-world purpose.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the update! I think we are using different anova functions. The header of mine is this: Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq) Also, I found why the discrepancy is there. The anova function is basically checking for all interaction coefficients as zero together and in the regression model, it checks individually. $\endgroup$
    – boY
    Commented Apr 7, 2017 at 17:43
  • 1
    $\begingroup$ @boY "All interaction coefficients", you say? Ah, I now see (thanks @Roland) that your IVs are categorical and hence the model P ~ A * B creates several interaction terms, not just one. Perhaps this is also why anova is using a $χ^2$ test instead of an $F$-test. $\endgroup$
    – Kodiologist
    Commented Apr 7, 2017 at 19:50
  • $\begingroup$ Yep. Strange that I didn't come across this in any text book. $\endgroup$
    – boY
    Commented Apr 7, 2017 at 20:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.