I'm unsure how to proceed to report my effects.
I have a glmer to run a logistic regression:
glmres.RU.conf.full = glmer(correct ~ choiceConf.scaled * condition + (1 + choiceConf.scaled|subject) + (1|cityPair), data = myDataForglm.RU, family = binomial)
I'm interested in two things: (1) the interaction between choiceConf.scaled * condition
and (2) the main effect of choiceConf.scaled
.
Testing my interaction effect doing a likelihood ratio test is straightforward:
glmres.RU.conf.sum = glmer(correct ~ choiceConf.scaled + condition + (1 + choiceConf.scaled|subject) + (1|cityPair), data = myDataForglm.RU, family = binomial)
anova(glmres.RU.conf.full, glmres.RU.conf.sum)
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
glmres.RU.conf.sum 7 4467.4 4513.5 -2226.7 4453.4
glmres.RU.conf.full 8 4469.3 4522.0 -2226.7 4453.3 0.0606 1 0.8056
I am unsure about what the correct null model to test the main effect of choiceConf.scaled
is. I can think of two approaches. First, I could compare a model including the two fixed effects vs. one including only the fixed factor I'm not interested in.
glmres.RU.conf.sum = glmer(correct ~ choiceConf.scaled + condition + (1 + choiceConf.scaled|subject) + (1|cityPair), data = myDataForglm.RU, family = binomial)
glmres.RU.null = glmer(correct ~ condition + (1 |subject) + (1|cityPair), data = myDataForglm.RU, family = binomial)
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
glmres.RU.null 4 4471.8 4498.2 -2231.9 4463.8
glmres.RU.conf.sum 7 4467.4 4513.5 -2226.7 4453.4 10.443 3 0.01515 *
Note that the AIC and BIC go in different directions. For the BIC, the null model is better (presumably because it's penalizing the 3 Df difference too strongly). So I tried to find a way to compare models that don't differ so much in complexity. I could do this by never considering the effect of condition in my model at all (I'm not interested in it for now, after all).
glmres.confOnly.full = glmer(correct ~ choiceConf.scaled + (1 + choiceConf.scaled|subject) + (1|cityPair), data = myDataForglm.RU, family = binomial)
glmres.confOnly.null = glmer(correct ~ 1 + (1 |subject) + (1|cityPair), data = myDataForglm.RU, family = binomial)
anova(glmres.confOnly.full, glmres.confOnly.null)
Df AIC BIC logLik deviance Chisq Chi Df Pr(>Chisq)
glmres.confOnly.null 3 5178.4 5198.2 -2586.2 5172.4
glmres.confOnly.full 6 5082.8 5122.4 -2535.4 5070.8 101.6 3 < 2.2e-16 ***
But this sounds incorrect to me. Is it ok to exclude factors (that are known to have a significant effect) from a model? Given that the BIC values are so different between the first and the second approach, I suspect it's not correct.
Any other comments welcome.