I have noticed a difference in post-hoc results produced from a repeated-measures ANOVA and a linear mixed model.

Here is reproducible code:


model <- lmer(Informed.liking ~ Gender * Product + (1|Consumer), data=ham, REML=T)
#Gender SS = 3.854
#Gender MS = 3.8544
#Gender F = 0.8789

ham$Gender.Product <- interaction(ham$Gender, ham$Product)
LMM.model.posthoc <- lmer(Informed.liking ~ Gender.Product + (1|Consumer), data=ham, REML=T)
summary(glht(LMM.model.posthoc, linfct = mcp(Gender.Product = "Tukey")), test = adjusted("none"))

#POST HOC COMPARISON OF 1.1 - 1.2: p = 0.191855
#POST HOC COMPARISON OF 1.2 - 1.3: p = 0.011224
#POST HOC COMPARISON OF 2.1 - 1.4: p = 0.671585
#POST HOC COPMARISON OF 1.3 - 2.2: p = 0.000209

summary(aov(Informed.liking ~ Gender * Product + Error(Consumer/Product), data=ham))
#Gender SS = 9.7
#Gender MS = 9.666
#Gender F = 0.879

anova <- aov(Informed.liking ~ Gender.Product, data=ham)
summary(LSD.test(anova, "Gender.Product", group=F, console=TRUE, p.adj=c("none")))

#POST HOC COMPARISON OF 1.1 - 1.2: p = 0.2312
#POST HOC COMPARISON OF 1.2 - 1.3: p = 0.0202
#POST HOC COMPARISON OF 1.4 - 2.1: p = 0.6480 
#POST HOC COPMARISON OF 1.3 - 2.2: p = 0.0001

The F-statistics are the same. However, the p-values from the post-hoc tests (Fisher's LSD in this example) differ, and I'm not sure why. I've noticed that SS and MS for Gender differ between the ANOVA and the LMM. As far as I can tell, I'm doing these analyses correctly.

In this example, is there a reason to prefer one method over the other? I would think these approaches should be equivalent here and I don't know why this isn't the case.

EDIT: This may be due to differences in p-value calculations for the packages. Post-hoc for the LMM uses glht from the Multcomp package, Post-hoc for ANOVA uses LSD.test from Agricolae. If this is the explanation, is there any reason to prefer one over the other in this situation? If not, what's the the explanation?

EDIT in response to design: In my situation, I have taken the same measurement on different parameters of individuals in different groups. Imagine I'm measuring blood concentration levels of different hormones from people of different ages.

In a repeated measures ANOVA, the parameter is the within-subjects effect (i.e. Hormone 1, Hormone 2, Hormone 3), and the groups are the between-subjects effect (i.e. Young, Middle-Age, Senior).

For a linear mixed model, the parameter and groups could be thought of as fixed effects, and individual as a random effect.

From my understanding, in this case both analyses would be equivalent. The differences in post-hoc p-values made me wonder if there's a difference in the approaches. I'm thinking now it's due to the differences in the packages and not in the analyses themselves, but I wanted to be sure.

Linear Mixed Model: concentration ~ parameter * group + (1|Individual)

Repeated Measures ANOVA: concentration ~ parameter * group + Error (Individual/Parameter)

  • $\begingroup$ A brief description of your design and your analysis model would be more useful to me than your code. Since two different sources produced the software, you should contact them both. Their formulae should be openly documented. Their algorithms (not their code) should be available for discussion with them. If no differences are identified initially then you might ask both to provide more detailed documentation. $\endgroup$ – David Smith Jun 6 '17 at 17:27
  • $\begingroup$ OK, I added a description of the general design. I'm thinking now the differences in the post-hoc tests are likely due to the package differences, since these analyses for this situation should be the same. $\endgroup$ – us-merul Jun 6 '17 at 18:07
  • $\begingroup$ I can't think why they would not be identical except for different algorithms (giving larger differences) or numerical errors due to a variety of differences in the programming (giving smaller differences). $\endgroup$ – David Smith Jun 6 '17 at 18:21
  • $\begingroup$ Thanks for your thoughts. This doesn't change what I would report, but I was curious about it in a broader sense. $\endgroup$ – us-merul Jun 6 '17 at 18:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.