How to interpret interaction becoming non-significant after removing one non-significant main effect from model?

Question

I am interested in an between-within interaction in a hierarchical logistic regression. I start my analysis with a model including all theoretically relevant predictors, including the between factor A, the within factor B, and their interaction AxB.

However, the main effect A is not significant, therefore I decide to only include main effect B and the interaction, which improves my model fit (BIC). However, in this new model, the interaction is not significant anymore (p = .0504), however main effect B is still highly significant. When adding another variable that greatly improves model fit, the interaction term is "even more non-significant" (p = .067).

How can I interpret this? Can I safely assume that the interaction effect I am interested in is simply not there and was an artifact in the earlier model? Why is it significant when including the non-significant main effect?

I am currently interpreting this as a power issue as the interaction effect has been shown in a previous study with a higher number of trials per participant.

Answer 1

It may be worth a quick review of interactions.

To keep things simple we will look at a linear model with an interaction between a continuous and a categorical (binary in this case) variable.

Suppose we have the following data:

Here both lines are parallel, so we don't expect to find an interaction:

> lm(y ~ x * g, data = dt) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  2.66783    0.29409   9.072 1.47e-14 ***
x            1.03295    0.04740  21.794  < 2e-16 ***
gB           2.51261    0.41590   6.041 2.89e-08 ***
x:gB        -0.05206    0.06703  -0.777    0.439    

And indeed we have no interaction. And from this output we can also say:

• The intercept for group A is 2.7
• The intercept for group B is 2.7 + 2.5 = 5.2
• The slope of x for both groups is 1

and this of course makes sense in terms of the plot.

Now, suppose we have this situation:

Here it is clear that the slopes of the 2 lines are not the same, hence we expect to find an interaction:

> lm(y ~ x * g, data = dt) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  2.97603    0.27725   10.73  < 2e-16 ***
x            1.02696    0.04468   22.98  < 2e-16 ***
gB           2.30562    0.39209    5.88 5.94e-08 ***
x:gB         1.43882    0.06319   22.77  < 2e-16 ***

And indeed we do. We can interpret this as follows:

• The intercept for group A is 3
• The intercept for group B is 3 + 2.3 = 5.3
• The slope of x for group A is 1
• The slope of x for group B is 1 + 1.4 = 2.4

Now, suppose we have:

Again it is abundantly clear that there is a difference in slope so we expect to find an interaction:

> lm(y ~ x * g, data = dt) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  2.97603    0.27725  10.734  < 2e-16 ***
x            0.02696    0.04468   0.603    0.548    
gB           2.30562    0.39209   5.880 5.94e-08 ***
x:gB         1.43882    0.06319  22.769  < 2e-16 ***

which indeed we do, but this time we also see that the main effect for x is zero, which of course makes sense because the line in the plot for A is flat, since the estimate for x tells us the association of y with x when g is at it's reference level (A).

But now suppose that we remove the main effect for x from the model, since it was "not significant":

> lm(y ~ g + x:g , data = dt) %>% summary()

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  2.97603    0.27725  10.734  < 2e-16 ***
gB           2.30562    0.39209   5.880 5.94e-08 ***
gA:x         0.02696    0.04468   0.603    0.548    
gB:x         1.46578    0.04468  32.804  < 2e-16 ***

So we can see that removing the main effect of x from the model is basically a reparamterisation of the previous model. Instead of getting a main effect of zero for x, which make sense because of the flat line for the group A, we have an additional interaction term gA:x, along with the interaction gB:x which is equivalent to the interaction estimate from the previous model. For this reason alone I think that removing a non-significant main effect from a model is a bad idea because it makes interpretation less intuitive. Also see here for other thoughts on why it is generally not a good idea to remove main effects when a variable is included in an interaction:
Logistic Regression Models Without Main Effects?
Including the interaction but not the main effects in a model
Do all interactions terms need their individual terms in regression model?