# When is it acceptable to probe an interaction effect between categorical variables? (Regression vs. anova)

This question seems to be an overly simple one but I would still appreciate an informed response.

I am testing the interaction effect between a categorical variable (5 levels) and a binary variable. Outcome variable is continuous.

I have completed by regression and anova to test for a possible interaction effect. Results of regression indicates a significant interaction effect between two groups (EX: class 1 and 2).

In short, the regression output suggests a significant interaction effect (Class=3 * BIN=1) and the anova output suggests that the SSQ of the interaction term is non-significant (CLASS:BIN).

(1) Does the significant interaction term (Class=3 * BIN=1) in the regression output justify "probing" of the significant interaction effect?

(2) What is the significance of the (non-significant) interaction term (CLASS:BIN) in anova?

(3) Say the moderator and DV are flipped (categorical by continuous predicting binary) so you are now running a binary logistic regression. P-value for the contrast of interaction effect for 2 dummy classes is still significant. Does it now make sense to probe the interaction solely within a regression framework? Would you not if the anova output is still saying the whole interaction term is NOT significant?

Regression output:

 Coef              S.E.   t       Pr(>|t|)
Intercept         0.6533 0.2402  2.72 0.0068
Class=2          -0.6955 0.2997 -2.32 0.0208
Class=3          -1.0229 0.2617 -3.91 0.0001
Class=4          -0.8226 0.2891 -2.84 0.0047
Class=5          -0.6086 0.3250 -1.87 0.0619
BIN=1             -0.3671 0.3082 -1.19 0.2343
Class=2 * BIN=1  0.7025 0.3955  1.78 0.0765
Class=3 * BIN=1  0.8602 0.3561  2.42 0.0162
Class=4 * BIN=1  0.6066 0.3743  1.62 0.1058
Class=5 * BIN=1  0.5961 0.4285  1.39 0.1649


Anova Table:

            SSQ     df1   df2     F value  Pr(>F)    eta2    partial.eta2
CLASS       19.04070  4  621.7598 4.0077   0.00322  0.04562   0.04726
BIN         7.11855   1 1014.6402 6.2523   0.01256  0.01706   0.01821
CLASS:BIN   7.40183   4  479.6875 1.3864   0.23751  0.01773   0.01892
Residual    383.82114  NA    NA      NA      NA      NA           NA

• The ANOVA table shows there is no interaction worthy of note. You are over-interpreting the regression as the individual coefficients do not tell you about the overall interaction but about specific parts of it. In the absence of an overall interaction they are only worth examining if you have a specific scientific hypothesis about them but since you do not give us that information we cannot tell. – mdewey Apr 30 '18 at 13:37

## 1 Answer

1) yes, it does justify probing the significance of the CLASS:BIN interaction with an ANOVA. However it is not entirely clear to me what does the term Class=3*BIN=1 represent in your table. Calling Y the outcome, is it the difference Y(Class=1,BIN=1)-Y(Class=3,BIN=1)?

2) the ANOVA performs instead a model comparison, asking whether the addition of the interaction term (CLASS:BIN) explained a large enough amount of variance in the data, with respect to the number of degrees of freedom it added (4)

based on the significance of the term Class=3*BIN=1 you cannot conclude that the effect of BIN on Class=1 and Class=3 differs. Indeed 2) suggests it probably does not. To test whether there is a difference you should build contrasts comparing the effect of BIN on each of the Classes. Then test for the significance of each contrast, keeping into account you are performing multiple tests and that you should correct for that. To help you more I should know exactly what do the coefficients in your table represent.

• You can try Bonferroni (but you have many tests, so probably nothing will stay significant) or FDR (false discovery rate), but in this latter case you should be careful about the interpretation. I use fdr correction in imaging data. There it does not matter whether each of the voxels I show as significant are indeed so. But with group comparisons, you will have to say some of the comparisons you say are significant maybe are not. This sounds confusing to me. – fabiob May 2 '18 at 9:39