1
$\begingroup$

I would like to test two hypotheses, but I am a little bit confused. I have a binary dependent variable z, my key variable a is also binary and three control variables b, c and d.

Now I want to test the following hypotheses.

1: If a is 1, then its more likely for z to be 1.

2: The effect of a on z decreases for smaller d

I have created two different model, m1 without interaction and m2 with interaction

m1 <- glm(z ~ a + b + c + d, 
                    data = data,
                    family = binomial(link = logit))
summary(m1)
Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.3588  -0.7265  -0.5430   0.8308   2.2636  

Coefficients:
                   Estimate Std. Error z value Pr(>|z|)    
(Intercept)        -2.59652    0.20205 -12.851  < 2e-16 ***
a                   0.45391    0.15227   2.981  0.00287 ** 
b                   2.05067    0.19366  10.589  < 2e-16 ***
c                   2.65482    0.22666  11.713  < 2e-16 ***
d                   0.14768    0.07242   2.039  0.04142 *  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    Null deviance: 1485.9  on 1243  degrees of freedom
Residual deviance: 1251.1  on 1239  degrees of freedom


m2 <- glm(dep ~ a + b + c + d + a:d, 
          data = data,
          family = binomial(link = logit))

summary(m2)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.3600  -0.7276  -0.5455   0.8237   2.2400  

Coefficients:
                           Estimate Std. Error z value Pr(>|z|)    
(Intercept)                -2.30656    0.24002  -9.610   <2e-16 ***
a                          -0.10448    0.30518  -0.342   0.7321    
b                           2.05082    0.19391  10.576   <2e-16 ***
c                           2.66973    0.22683  11.770   <2e-16 ***
d                          -0.04204    0.11547  -0.364   0.7158    
a:d                         0.31179    0.14809   2.105   0.0353 *  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


    Null deviance: 1485.9  on 1243  degrees of freedom
Residual deviance: 1246.7  on 1238  degrees of freedom


Analysis of Deviance Table

Model 1: z ~ a + b + c + 
    d
Model 2: z ~ a + b + c + 
    d + a:d
  Resid. Df Resid. Dev Df Deviance Pr(>Chi)    
1      1239     1251.1                         
2      1238     1232.5  1   18.688 1.54e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Model 1 shows that the first hypothesis is true and a is significant. However, in model 2 the coefficient a is no longer significant and we can see negative sign.

Can I say that hypothesis 1 is true, although other models show a different result? Or does hypothesis 1 to be true among all models?

$\endgroup$
  • $\begingroup$ Which of your two models is best supported by your data? Can you run the command anova(m1,m2) and update your question with the output it produces? If the p-value produced by anova is small (e.g., < 0.05), then m2 is more appropriate for your data and should be used as a basis for addressing your hypotheses. Once you update us on the anova p-value, we can comment more pertinently on how you should proceed. $\endgroup$ – Isabella Ghement Dec 4 '18 at 15:12
  • $\begingroup$ Thank you. I added the output of anova(m1,m2), but I don't see a p-value. $\endgroup$ – Jensxy Dec 4 '18 at 16:25
  • $\begingroup$ Can you try this: anova(m1, m2, test="LRT") ? This should enable you to compare the two models via a likelihood ratio test and should produce a p-value. See stats.stackexchange.com/questions/59879/… for an example. Then you can post the updated anova output with a p-value in your question. Thanks! $\endgroup$ – Isabella Ghement Dec 4 '18 at 16:49
  • 1
    $\begingroup$ Okay, I added the p-values. Just for my understanding. If the p-value in the output is <0.05 then the second model in anova(model 1, model 2) is preferable? $\endgroup$ – Jensxy Dec 4 '18 at 16:55
3
$\begingroup$

So the log likelihood ratio test showed that your second model is a better fit than your first model. You should interpret that one.

Firstly, is d or a categorical or continuous? If either of them are categorical then you should plot your results as they really help you to understand your results. Suppose that both of them are, our interaction term a:d is significant so you should expect the mean of your dependent variable z to be different at different levels of a to not be parallel across d, they may even intersect e.g like this. If they are both continuous, it's hard to plot them, but it's still the same interpretation.

With regards to your hypothesis, if the effect of a:d is significant it means that the effect of a on z depends on the levels/size of d. You'll see below it's in the direction you hypothesized. When there's an interaction term its effect 'eats up' all the effect of the main effects a and d so that's why they can end up being non significant sometimes. Often, people are told not to interpret the main effects anymore if the interaction is significant. The main effect is still included in the model because the interaction's effect is nested within the main effect. You still need the main effect to, for example, predict values of the dependent variable (see below). But for hypothesis testing purposes it often no longer makes sense to interpret the main effect on its own anymore. Because there is no 'single' main effect of a if the interaction is significant. As your hypothesis shows, if the effect of a on z decreases as d decreases, so there's no way to talk about a on its own without talking about d also. Again, there isn't a single/unified effect of a, but an changing effect, depending on d.

Now, for balance, there is a counterpoint to this common perception here, where they say sometimes it's ok to interpret a even if the interaction term is significant. But all they're saying is that sometimes/occasionally even if the main effect of a changes across the level of d you can still conclude a is always a positive effect. But I think that's largely quibbling, because a still has a different effect across d even if it's always a positive effect. The coefficient of a still doesn't make sense on its own.

Finally, in your case. Let's have a look at your final model equation (i'm going to assume that they're continuous now, because it just makes it easier to explain, but if they are categorical then just look at this website for more info): $$\frac{p}{1-p}=e^{-2.30656-0.10448a+2.05082b+2.66973c-0.04204d+0.31179a*d}$$

So, now control for b and c and the intercept by substituting in values of 0 for them.

$$\frac{p}{1-p}=e^{-0.10448a-0.04204d+0.31179a*d}$$

For when a is 1 but d is large, let's say 5 $$\frac{p}{1-p}=e^{-0.10448*1-0.04204*5+0.31179*1*5}=3.4704$$ which means the odds of z has increased by 247%. Now, for a 1 unit increase in a and when d is now small, let's say 1, sub in those numbers $$\frac{p}{1-p}=e^{-0.10448*1-0.04204*1+0.31179*1*1}=1.1797$$ which means the odds of z has increased by only 17%. So, your second hypothesis is true, for smaller values of d the effect of a has decreased.

$\endgroup$
  • $\begingroup$ Thank you very much. Yes, a is binary. However, my teacher told me that I should use 1 for the intercept and the mean of b and c to control for them. I am confused. $\endgroup$ – Jensxy Dec 4 '18 at 17:33
  • $\begingroup$ ok, firstly, be careful the interpretation is now the odds ratio between a and not a, not straight up odds. So the odds ratio is the ratio between the odds of z given a and the odds of z given not a. I'm not sure why your teacher would want to control that way, but they must have their reasons. So, all you do is sub in those values, the odds ratio between a and not a given a 1 unit increase in d when b=mean and c=mean and intercept=1 is XXX. $\endgroup$ – Huy Pham Dec 4 '18 at 17:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.