2
$\begingroup$

I encountered a case where lme gives strange results. I have data gathered using repeated measures. I wanted to check if my predictor would influence the data. I built a model using lme, which gave me significant results. But afterwards, I saw that my data is each category of the predictor was identical for each subject. Strangely, lmer gives me the correct answer (p value = 1). So what is the difference between the two functions? Is it a case not handled by lme?

Here is a small example to reproduce the problem :

test = data.frame(Subject = as.factor(c(1,1,2,2,3,3)), 
     Condition = as.factor(c(1,2,1,2,1,2)), Data = c(5,5,6,6,2,2))
testModel = lme(Data ~ Condition, data = test, 
       random = ~1|Subject)
Anova(testmodel)

The output is: Chisq(1) = 2.94, p = 0.086 (no warning)

testModel = lmer(Data ~ Condition + (1|Subject), data = test)
Anova(testmodel)

The output is: Chisq(1) = 0, p = 1 and there is a warning.

I tested using both Anova and anova, using "ML" and "REML" options, it remains the same.

$\endgroup$
3
  • $\begingroup$ The model is fine, your data is not, every condition has the exact same values that's why all the coefficients except intercept are equal to 0. $\endgroup$ Sep 27, 2018 at 10:14
  • $\begingroup$ I agree with @user2974951. Because there is no residual variance in this situation the model cannot be identified. lmer actually shows this in warning message "Model is nearly unidentifiable". You can also see this in the estimated value of the residual variance. As there is no variance on the lowest level, it is probably better to aggregate the data in this situation $\endgroup$
    – Niek
    Sep 27, 2018 at 10:38
  • 2
    $\begingroup$ Thank you for the answer. However, as I said I know that there is something wrong with my data, it is only an example, the problem is not there. My question was more on why does lme give no warning about it, and why does lmer give the right answer, contrary to lme. What changes in their computation? $\endgroup$
    – Pyxel
    Sep 27, 2018 at 11:33

1 Answer 1

5
$\begingroup$

While they fit the same models, the computational machinery of lme and lmer is almost completely different. That means that for pathological problems (like this one) they can give different answers.

Here's another view of the lme results:

> coef(summary(testModel))
                    Value    Std.Error DF   t-value    p-value
(Intercept)  4.333333e+00 1.201848e+00  2  3.605558 0.06905044
Condition2  -7.327098e-16 5.233641e-16  2 -1.400000 0.29647350

For Condition2, both the estimate and the standard error are tiny; however, the estimate isn't exactly zero, so the Wald chi-squared test from car::Anova() gives a (more or less arbitrary) answer (on my system it gives Chisq=1.96, p=0.1615 - different from your answer but that's to be expected because we're basically looking at noise).

Results from lme4:

> coef(summary(testModel2))
            Estimate   Std. Error  t value
(Intercept) 4.571429 9.239712e-01 4.947588
Condition2  0.000000 8.412536e-08 0.000000

Here Condition2 is estimated as exactly zero, so the chi-squared value is 0 and the p-value is 1.

It would be nice if the packages could detect these pathologies better, but there are so many possible problems that they can't all be detected automatically.

Responding to comment:

if I did not look carefully at the data (someone else sent it to me), I could have missed it and report wrong results.

In this case, you can also tell there's something weird going on if you look at the coefficients of the model instead of going straight to the Anova() results - you can see that the estimated coefficient is really, really tiny. This is admittedly a gray area - the more safeguards in the software the better, and there are certainly cases I've seen where software failing to warn about easily detectable problems rises nearly to the level of a bug - but at the risk of sounding preachy, it's always the analyst's responsibility (whether it's your data or someone else's) to look at the data, and the results of the analysis, carefully ...

$\endgroup$
3
  • $\begingroup$ Thank you for your complete answer (and your edit)! I did not like this kind of problem because if I did not look carefully at the data (someone else sent it to me), I could have missed it and report wrong results. So the "noise" we are looking at is introduced during the computation right? And for more traditional cases, this noise would be completely trampled by predictor's influence? $\endgroup$
    – Pyxel
    Sep 27, 2018 at 14:38
  • 1
    $\begingroup$ Yes. See edits ... $\endgroup$
    – Ben Bolker
    Sep 27, 2018 at 16:06
  • 1
    $\begingroup$ I agree, as you said, it is our responsibility to sufficiently look into the data and the results to be sure about the results. Plotting data and looking at coefficient is easy enough for anyone to look. I will be more careful next time, thanks! $\endgroup$
    – Pyxel
    Sep 27, 2018 at 16:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.