Model fit is improved ONLY by random effects in linear mixed effects model

Question

I am trying to evaluate fixed effects by model comparison using lme4. Every time I add fixed effect, I also add corresponding random intercept and slope. When I compare a model with fixed effects (m1) vs. null model (m0), I see improvement in the model fit. However, it seems that the improvement is achieved only by random slopes, i.e. if I leave only random intercept in my model (m1a), there is no significant difference between m1a and m0.

m0 <- lmer(dv ~ 1 + (1|id), data = df, REML=F)

m1 <- lmer(dv ~ 1 + A + (1+A|id), data = df, REML=F)

m1a <- lmer(dv ~ 1 + A + (1|id), data = df, REML=F)

anova(m0, m1) # p < 0.05

anova(m0, m1a) # p > 0.05

My question is how should I interpret these results? The effect of A is not significant, however, the variation in this effect between participants seems to explain some variance.

Answer 1

These results indicate that there is very little overall "effect" of A But there is appreciable variation in A between subjects.

Edit to address the further question of how to proceed.

You have two options.

The first is to remove the fixed effect for A and the second is to retain it.

I don't like to make decisions based on p-values so I would tend to retain it unless I had good a priori reasons for thinking that the overall effects should be zero. It might be that your sample size was insufficient to detect a meaningful fixed effect for A. It might also be that this particular sample is not representative of the population. A lot will depend on your research goals.

But before doing anything further it would be a good idea to actually plot your data, and this will give a good idea of what is going on.

Answer 2

This is a lot easier if you make each model explicit. I'll assume id refers to participants.

• m0: Intercepts differ between participants, but the effect of A is the same for all participants, and is 0.
• m1: Intercepts and the effect of A differ between participants, and the average effect of A across across participants (the fixed effect) is not necessarily 0.
• m1a: Intercepts differ between participants, the effect of A is the same for all participants, and the effect of A is not necessarily 0.

m1 is significantly better than m0, meaning that either the overall effect of A isn't zero, or the effect of A isn't the same for every participant.

m1a isn't significantly better than m0, meaning that your data are consistent with the overall effect of A being zero.

Therefore, it's probably the case that while the overall effect of A is zero, some participants have positive effects, and some have negative effects. This suggests the best model overall would actually be

lmer(dv ~ 1 + (1 + A|id), data = df, REML=F)