# Why do I get a massive drop in AIC/BIC when adding a main effect that isn't even significant?

I have no explanation for this. Note how "school" isn't significant, yet the model with only that main effect has a much lower AIC (and BIC) than the one with time, intervention, and intervention*time (2 of those are significant).

Response is binary: 0=no, 1=yes

There are three predictors:

School: 4 levels (2 schools got treatment, 2 didn't)

Time: baseline, timepoint 1, timepoint 2

There are ~200 observations per treatment per time point (200*2 treatments*3 times = 1200 total n).

Random effect is SID (student ID#)

Mixed modelling:

model1 <- glmer(y ~ (1 | studentid),
data = dat, family = binomial, na.action=na.exclude)
AIC: 1314
Fixed effects:
Estimate Std. Error z value Pr(>|z|)
(Intercept)  -1.0759     0.1316  -8.178  2.9e-16 ***

model2 <- glmer(y ~ school.factor +
(1 | studentid), data = dat, family = binomial, na.action=na.exclude)
AIC: 462
Fixed effects:
Estimate Std. Error z value Pr(>|z|)

(Intercept)                             -0.71668    0.26635  -2.691  0.00713 **
school.factorHorace Elementary School   -0.46895    0.43541  -1.077  0.28147
school.factorLegacy Elementary School    0.01046    0.33193   0.032  0.97486
school.factorTea Area Legacy Elementary -0.18038    0.32310  -0.558  0.57665

model3 <- glmer(y ~ intervention + time + time*intervention +
(1 | studentid), data = dat, family = binomial)
AIC: 1314
Fixed effects:
Estimate Std. Error z value Pr(>|z|)
(Intercept)        -1.26999    0.25739  -4.934 8.05e-07 ***
intervention       -0.04564    0.33241  -0.137  0.89080
time2               0.31367    0.28942   1.084  0.27846
time3               0.78005    0.29532   2.641  0.00826 **
intervention:time2 -0.07410    0.39286  -0.189  0.85040
intervention:time3 -0.77871    0.39605  -1.966  0.04928 *


For m3 the AIC is slightly worse if we drop any of the 3 main effects. Also, if I try to include both school.factor and any other main effect variable I get a warning that the fixed-effect matrix is rank deficient so they drop the additional columns/coefficients, always just leaving me with school.factor.

Any idea what's going on here?

• Can you try plotting your data? I suspect that there might be difference due to school ID that are very strong and potentially tower over the treatment effect. For example, school will be much better in capturing socio-economic indicators that might affect a student-associated outcome. Commented Aug 7, 2018 at 20:50