0
$\begingroup$

I am interested in an between-within interaction in a hierarchical logistic regression. I start my analysis with a model including all theoretically relevant predictors, including the between factor A, the within factor B, and their interaction AxB.

However, the main effect A is not significant, therefore I decide to only include main effect B and the interaction, which improves my model fit (BIC). However, in this new model, the interaction is not significant anymore (p = .0504), however main effect B is still highly significant. When adding another variable that greatly improves model fit, the interaction term is "even more non-significant" (p = .067).

How can I interpret this? Can I safely assume that the interaction effect I am interested in is simply not there and was an artifact in the earlier model? Why is it significant when including the non-significant main effect?

I am currently interpreting this as a power issue as the interaction effect has been shown in a previous study with a higher number of trials per participant.

$\endgroup$
3
  • $\begingroup$ It is unusual, most if the time lacking sense, to retain an interaction while one of the main effects making it up is dropped. If there is no A in the model, A*B no longer has the meaning of the interaction, rather, it is a new predictor, just correlated with B. $\endgroup$
    – ttnphns
    Commented Jun 25, 2021 at 10:15
  • $\begingroup$ Would it then be justified to keep both main effects and the interaction even though one main effect is not significant? Or would you remove the interaction and the main effect all together? ..the BIC basically answers this for me, I would have to remove both of it. $\endgroup$
    – Max J.
    Commented Jun 25, 2021 at 10:29
  • 1
    $\begingroup$ That decision is on you. Sometimes significant interactions are important and can be interpreted while a main effect behind it is nonsignigicant $\endgroup$
    – ttnphns
    Commented Jun 25, 2021 at 10:31

1 Answer 1

2
$\begingroup$

It may be worth a quick review of interactions.

To keep things simple we will look at a linear model with an interaction between a continuous and a categorical (binary in this case) variable.

Suppose we have the following data:

enter image description here

Here both lines are parallel, so we don't expect to find an interaction:

> lm(y ~ x * g, data = dt) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  2.66783    0.29409   9.072 1.47e-14 ***
x            1.03295    0.04740  21.794  < 2e-16 ***
gB           2.51261    0.41590   6.041 2.89e-08 ***
x:gB        -0.05206    0.06703  -0.777    0.439    

And indeed we have no interaction. And from this output we can also say:

  • The intercept for group A is 2.7
  • The intercept for group B is 2.7 + 2.5 = 5.2
  • The slope of x for both groups is 1

and this of course makes sense in terms of the plot.

Now, suppose we have this situation:

enter image description here

Here it is clear that the slopes of the 2 lines are not the same, hence we expect to find an interaction:

> lm(y ~ x * g, data = dt) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  2.97603    0.27725   10.73  < 2e-16 ***
x            1.02696    0.04468   22.98  < 2e-16 ***
gB           2.30562    0.39209    5.88 5.94e-08 ***
x:gB         1.43882    0.06319   22.77  < 2e-16 ***

And indeed we do. We can interpret this as follows:

  • The intercept for group A is 3
  • The intercept for group B is 3 + 2.3 = 5.3
  • The slope of x for group A is 1
  • The slope of x for group B is 1 + 1.4 = 2.4

Now, suppose we have:

enter image description here

Again it is abundantly clear that there is a difference in slope so we expect to find an interaction:

> lm(y ~ x * g, data = dt) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  2.97603    0.27725  10.734  < 2e-16 ***
x            0.02696    0.04468   0.603    0.548    
gB           2.30562    0.39209   5.880 5.94e-08 ***
x:gB         1.43882    0.06319  22.769  < 2e-16 ***

which indeed we do, but this time we also see that the main effect for x is zero, which of course makes sense because the line in the plot for A is flat, since the estimate for x tells us the association of y with x when g is at it's reference level (A).

But now suppose that we remove the main effect for x from the model, since it was "not significant":

> lm(y ~ g + x:g , data = dt) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  2.97603    0.27725  10.734  < 2e-16 ***
gB           2.30562    0.39209   5.880 5.94e-08 ***
gA:x         0.02696    0.04468   0.603    0.548    
gB:x         1.46578    0.04468  32.804  < 2e-16 ***

So we can see that removing the main effect of x from the model is basically a reparamterisation of the previous model. Instead of getting a main effect of zero for x, which make sense because of the flat line for the group A, we have an additional interaction term gA:x, along with the interaction gB:x which is equivalent to the interaction estimate from the previous model. For this reason alone I think that removing a non-significant main effect from a model is a bad idea because it makes interpretation less intuitive. Also see here for other thoughts on why it is generally not a good idea to remove main effects when a variable is included in an interaction:
Logistic Regression Models Without Main Effects?
Including the interaction but not the main effects in a model
Do all interactions terms need their individual terms in regression model?

$\endgroup$
2
  • $\begingroup$ Super helpful and great visualisation! Thank you very much. $\endgroup$
    – Max J.
    Commented Jun 25, 2021 at 14:01
  • $\begingroup$ You're welcome :) Glad it can help ! $\endgroup$ Commented Jun 25, 2021 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.