3
$\begingroup$

I am trying to evaluate fixed effects by model comparison using lme4. Every time I add fixed effect, I also add corresponding random intercept and slope. When I compare a model with fixed effects (m1) vs. null model (m0), I see improvement in the model fit. However, it seems that the improvement is achieved only by random slopes, i.e. if I leave only random intercept in my model (m1a), there is no significant difference between m1a and m0.

m0 <- lmer(dv ~ 1 + (1|id), data = df, REML=F)

m1 <- lmer(dv ~ 1 + A + (1+A|id), data = df, REML=F)

m1a <- lmer(dv ~ 1 + A + (1|id), data = df, REML=F)

anova(m0, m1) # p < 0.05

anova(m0, m1a) # p > 0.05

My question is how should I interpret these results? The effect of A is not significant, however, the variation in this effect between participants seems to explain some variance.

$\endgroup$
4
$\begingroup$

These results indicate that there is very little overall "effect" of A But there is appreciable variation in A between subjects.

Edit to address the further question of how to proceed.

You have two options.

The first is to remove the fixed effect for A and the second is to retain it.

I don't like to make decisions based on p-values so I would tend to retain it unless I had good a priori reasons for thinking that the overall effects should be zero. It might be that your sample size was insufficient to detect a meaningful fixed effect for A. It might also be that this particular sample is not representative of the population. A lot will depend on your research goals.

But before doing anything further it would be a good idea to actually plot your data, and this will give a good idea of what is going on.

$\endgroup$
2
  • $\begingroup$ Hi Robert! Thanks! That's what I understand. I just don't understand what should be the next step in the analysis. If there's appreciable variation in A between subjects, I guess, I shouldn't drop this fixed effect, and maybe should try to find a source of this variation, e.g. some coasters of participants... $\endgroup$ Oct 5 '20 at 11:58
  • $\begingroup$ I've updated my answer $\endgroup$ Oct 5 '20 at 12:58
4
$\begingroup$

This is a lot easier if you make each model explicit. I'll assume id refers to participants.

  • m0: Intercepts differ between participants, but the effect of A is the same for all participants, and is 0.
  • m1: Intercepts and the effect of A differ between participants, and the average effect of A across across participants (the fixed effect) is not necessarily 0.
  • m1a: Intercepts differ between participants, the effect of A is the same for all participants, and the effect of A is not necessarily 0.

m1 is significantly better than m0, meaning that either the overall effect of A isn't zero, or the effect of A isn't the same for every participant.

m1a isn't significantly better than m0, meaning that your data are consistent with the overall effect of A being zero.

Therefore, it's probably the case that while the overall effect of A is zero, some participants have positive effects, and some have negative effects. This suggests the best model overall would actually be

lmer(dv ~ 1 + (1 + A|id), data = df, REML=F)
$\endgroup$
1
  • $\begingroup$ Hi! Thanks for your reply. I guess, in this case, the most reasonable next step would be to look at the individual data and to figure out what makes some participants show the effect and others show the opposite effect. $\endgroup$ Oct 5 '20 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.