Interpretation of different logistic regression models to test hypotheses

Question

I would like to test two hypotheses, but I am a little bit confused. I have a binary dependent variable z, my key variable a is also binary and three control variables b, c and d.

Now I want to test the following hypotheses.

1: If a is 1, then its more likely for z to be 1.

2: The effect of a on z decreases for smaller d

I have created two different model, m1 without interaction and m2 with interaction

m1 <- glm(z ~ a + b + c + d, 
                    data = data,
                    family = binomial(link = logit))
summary(m1)
Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.3588  -0.7265  -0.5430   0.8308   2.2636  

Coefficients:
                   Estimate Std. Error z value Pr(>|z|)    
(Intercept)        -2.59652    0.20205 -12.851  < 2e-16 ***
a                   0.45391    0.15227   2.981  0.00287 ** 
b                   2.05067    0.19366  10.589  < 2e-16 ***
c                   2.65482    0.22666  11.713  < 2e-16 ***
d                   0.14768    0.07242   2.039  0.04142 *  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    Null deviance: 1485.9  on 1243  degrees of freedom
Residual deviance: 1251.1  on 1239  degrees of freedom


m2 <- glm(dep ~ a + b + c + d + a:d, 
          data = data,
          family = binomial(link = logit))

summary(m2)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.3600  -0.7276  -0.5455   0.8237   2.2400  

Coefficients:
                           Estimate Std. Error z value Pr(>|z|)    
(Intercept)                -2.30656    0.24002  -9.610   <2e-16 ***
a                          -0.10448    0.30518  -0.342   0.7321    
b                           2.05082    0.19391  10.576   <2e-16 ***
c                           2.66973    0.22683  11.770   <2e-16 ***
d                          -0.04204    0.11547  -0.364   0.7158    
a:d                         0.31179    0.14809   2.105   0.0353 *  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


    Null deviance: 1485.9  on 1243  degrees of freedom
Residual deviance: 1246.7  on 1238  degrees of freedom


Analysis of Deviance Table

Model 1: z ~ a + b + c + 
    d
Model 2: z ~ a + b + c + 
    d + a:d
  Resid. Df Resid. Dev Df Deviance Pr(>Chi)    
1      1239     1251.1                         
2      1238     1232.5  1   18.688 1.54e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Model 1 shows that the first hypothesis is true and a is significant. However, in model 2 the coefficient a is no longer significant and we can see negative sign.

Can I say that hypothesis 1 is true, although other models show a different result? Or does hypothesis 1 to be true among all models?

Answer 1

So the log likelihood ratio test showed that your second model is a better fit than your first model. You should interpret that one.

Firstly, is d or a categorical or continuous? If either of them are categorical then you should plot your results as they really help you to understand your results. Suppose that both of them are, our interaction term a:d is significant so you should expect the mean of your dependent variable z to be different at different levels of a to not be parallel across d, they may even intersect e.g like this. If they are both continuous, it's hard to plot them, but it's still the same interpretation.

With regards to your hypothesis, if the effect of a:d is significant it means that the effect of a on z depends on the levels/size of d. You'll see below it's in the direction you hypothesized. When there's an interaction term its effect 'eats up' all the effect of the main effects a and d so that's why they can end up being non significant sometimes. Often, people are told not to interpret the main effects anymore if the interaction is significant. The main effect is still included in the model because the interaction's effect is nested within the main effect. You still need the main effect to, for example, predict values of the dependent variable (see below). But for hypothesis testing purposes it often no longer makes sense to interpret the main effect on its own anymore. Because there is no 'single' main effect of a if the interaction is significant. As your hypothesis shows, if the effect of a on z decreases as d decreases, so there's no way to talk about a on its own without talking about d also. Again, there isn't a single/unified effect of a, but an changing effect, depending on d.

Now, for balance, there is a counterpoint to this common perception here, where they say sometimes it's ok to interpret a even if the interaction term is significant. But all they're saying is that sometimes/occasionally even if the main effect of a changes across the level of d you can still conclude a is always a positive effect. But I think that's largely quibbling, because a still has a different effect across d even if it's always a positive effect. The coefficient of a still doesn't make sense on its own.

Finally, in your case. Let's have a look at your final model equation (i'm going to assume that they're continuous now, because it just makes it easier to explain, but if they are categorical then just look at this website for more info): $$\frac{p}{1-p}=e^{-2.30656-0.10448a+2.05082b+2.66973c-0.04204d+0.31179a*d}$$

So, now control for b and c and the intercept by substituting in values of 0 for them.

$$\frac{p}{1-p}=e^{-0.10448a-0.04204d+0.31179a*d}$$

For when a is 1 but d is large, let's say 5 $$\frac{p}{1-p}=e^{-0.10448*1-0.04204*5+0.31179*1*5}=3.4704$$ which means the odds of z has increased by 247%. Now, for a 1 unit increase in a and when d is now small, let's say 1, sub in those numbers $$\frac{p}{1-p}=e^{-0.10448*1-0.04204*1+0.31179*1*1}=1.1797$$ which means the odds of z has increased by only 17%. So, your second hypothesis is true, for smaller values of d the effect of a has decreased.