Different results doing mixed effects anova and post hoc tests of simple effects and paired T-test

I have run a mixed effects anova in r to look at the performance differences in two groups (control, patient) in 5 separate conditions. The conditions are my within subject factors. Group is the between subject factor. Performance is the dependent variable.

mixed.anova <- aov(performance ~ group * conditions +  Error(ID/conditions),
data=volumes.rm )
summary( mixed.anova)


There was an interaction between condition and group. I ran post-hoc tests of simple effects with the multcomp and nlme package:

volumes.rm <- within(volumes.rm,
{interaction.object <- interaction(group, condition)})

rm.anova <- lme(volume ~ interaction.object, random=~1|ID, data=volumes.rm)
summary(glht(rm.anova, linfct=mcp(interaction.object="Tukey")))


Three of my conditions are significantly different between groups.

I then look at the data a different way using paired T-tests. I look a group differences in performance in the six separate conditions using separate paired t tests. One of the conditions that wasn't significant in the mixed effects anova becomes significant now.

How can I reconcile the different results using the different tests?

First, you should confirm that your p-values are still close between the two methods. If your p-values are near the decision threshold (e.g. around 0.05), then claims of significance might change even if your p-values only changed from 0.04 to 0.055. If your p-values are profoundly different, then you should make sure the assumptions of the mixed model are met (see below).

Here are the things that could produce differences in significance, from likely most important to least:

1. The default in multcomp is to adjust for multiple comparisons using the sigle-step method. You have to include the argument test=adjusted("none") in the summary function to prevent this (and make comparable to running individual t-tests).
2. Using the glht with the full ANOVA model will use the same estimate of variance across all of the different comparisons (i.e. a pooled variance; see how the standard errors are the same in the summary.glht output?). This is the strength of this kind of analysis—it allows you to make more precise estimates of each comparison. It is important that the variance in the model is homoscedastic otherwise it would be invalid to pool variance across the different tests. Because the data are never perfectly homoscedastic, the p-values between t-tests and the pooled tests will change depending upon how far the truth varies from the estimated variance.
3. Multcomp uses z-statistics with linear mixed models and not t-statistics.

Edit Yes, you can specify the tests you want. Here's a working example. It also shows how the results are similar with paired t tests. I have given an example testing the differences between within-group means, but you can just run the comparisons you are interested in. Also delete the test=adjusted("none") or change the methods to adjust for multiple comparisons.

# Create a reproducible example - 2 x 2 between-by-within experiment
set.seed(42)
myData=data.frame(y=rnorm(100, 0, 2), A=gl(2, 50, labels=c("a1", "a2")), B=c(gl(2, 25, labels=c("b1", "b2")), gl(2, 25, labels=c("b1", "b2"))))
myData$B=as.factor(ifelse(myData$B==1, "b1", "b2"))
myData$B=relevel(myData$B, ref="b1")
myData$id=rep(1:50, 2) # A is between-subject factor, B is within-subject factor # Run the ANOVA mixed.anova <- aov(y ~ A * B + Error(id/B),data = myData) # Put 2 factors into one to allow cell-means coding myData <- within(myData, {interaction.object <- interaction(A, B)}) # Run LME rm.anova <- lme(y ~ interaction.object , random=~1|id, data=myData) # We can run multcomp to get simple effects using pooled variance (comparing b1 vs b2 in a1 then in a2) summary(glht(rm.anova, linfct=mcp(interaction.object="Tukey")), test=adjusted("none")) # gives 0.130 and 0.776 # We can run multcomp on only our contrats of interest (note they are the same as when all the contrasts are tested because we are not adjusting for multiple comparisons) myContrasts=c("a1.b1 - a1.b2 = 0", "a2.b1 - a2.b2 = 0") summary(glht(rm.anova, linfct=mcp(interaction.object=myContrasts)), test=adjusted("none")) # gives 0.130 and 0.776 # Test b1 vs b2 in a1 and then in a2 as individual t-tests t.test(y ~ B, data=myData[myData$A=="a1",], paired=TRUE) # 0.195
t.test(y ~ B, data=myData[myData\$A=="a2",], paired=TRUE) # 0.735

• It looks like the issue is that mult comp is doing all possible pairwise comparisons. ie: it is comparing condition 1 to conditions 2-5 within each group. I would like to be able to specify the pairwise comparisons to make. Is there a way to specify to compare only conditions between groups? I want to make only 5 pairwise comparisons. ie: condition 1 between group, condition 2 between groups, condition 3 between groups, condition 4 between groups, condition 5 between groups. Thanks so much. – lexi Oct 11 '16 at 13:06