# Big difference between a t-test and a F-test in a mixed model (anova vs summary in lmerTest)

While helping someone else with their analyses, I've run into a question regarding the difference between t-tests and F-tests for linear mixed models in lme4 for R, as provided by lmerTest. I'm aware of the problems with calculating any kind of p-values for linear mixed models (as I understand, primarily due to the fact that definition of the degrees of freedom is problematic), as well as the problems with interpreting main effects in the presence of significant interactions (based on the marginality principle).

Briefly, the data are from an experiment with two conditions (congruity TRUE/FALSE), measured on six sets of sensors which can be described as a combination of two factors: anteriority (anterior/posterior) and laterality (left/central/right).

As can be seen from the summary output below, the t.tests do not show a significant congruity effect (p = 0.12), while the anova output shows a very significant congruity effect (p = 2.8e-10). Since congruity has only two levels, this cannot be the result of the F-test doing an omnibus test over several levels of the fixed factor. I am therefore unsure what causes the very significant result in the anova output. Is this due to the fact that there are strong interactions involving congruity which of course depend on the inclusion of the main effect in the model parametrization?

I have looked for a previous answer to this question on CrossValidated but I have not been able to find anything relevant except possibly the first answer to this question. However, if that does provide a real answer then it is implicit in the mathematics, and I am looking for a conceptual answer that I can explain to the person I am trying to help.

> final.mod<-lmer(uV~1+factor(congruity)*factor(laterality)*factor(anteriority)+(1|sent.id)+(1|Subject),data=selected.data)
> summary(final.mod)
Linear mixed model fit by REML

t-tests use  Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: uV ~ 1 + factor(congruity) * factor(laterality) * factor(anteriority) +      (1 | sent.id) + (1 | Subject)
Data: selected.data
REML criterion at convergence: 348903.5
Scaled residuals:
Min      1Q  Median      3Q     Max
-7.0440 -0.6002  0.0069  0.6038 11.3912
Random effects:
Groups   Name        Variance Std.Dev.
sent.id  (Intercept)   1.773   1.332
Subject  (Intercept)   2.548   1.596
Residual             111.396  10.554
Number of obs: 46176, groups:  sent.id, 41; Subject, 30
Fixed effects:
Estimate Std. Error         df t value Pr(>|t|)
(Intercept)                                                                 4.768e-03  3.973e-01  7.900e+01   0.012   0.9905
factor(congruity)TRUE                                                       3.758e-01  2.410e-01  4.611e+04   1.559   0.1189
factor(laterality)left                                                      7.154e-02  2.430e-01  4.610e+04   0.294   0.7685
factor(laterality)right                                                    -2.003e-01  2.430e-01  4.610e+04  -0.824   0.4098
factor(anteriority)posterior                                               -4.203e-02  2.430e-01  4.610e+04  -0.173   0.8627
factor(congruity)TRUE:factor(laterality)left                               -1.013e-01  3.404e-01  4.610e+04  -0.298   0.7660
factor(congruity)TRUE:factor(laterality)right                               7.233e-02  3.404e-01  4.610e+04   0.213   0.8317
factor(congruity)TRUE:factor(anteriority)posterior                          6.162e-01  3.404e-01  4.610e+04   1.810   0.0702 .
factor(laterality)left:factor(anteriority)posterior                         2.568e-01  3.437e-01  4.610e+04   0.747   0.4549
factor(laterality)right:factor(anteriority)posterior                        1.763e-01  3.437e-01  4.610e+04   0.513   0.6080
factor(congruity)TRUE:factor(laterality)left:factor(anteriority)posterior  -5.162e-02  4.813e-01  4.610e+04  -0.107   0.9146
factor(congruity)TRUE:factor(laterality)right:factor(anteriority)posterior -2.420e-01  4.813e-01  4.610e+04  -0.503   0.6152
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Correlation of Fixed Effects:
(Intr) fc()TRUE fctr(ltrlty)l fctr(ltrlty)r fctr(n) fctr(cngrty)TRUE:fctr(ltrlty)l fctr(cngrty)TRUE:fctr(ltrlty)r
fctr(c)TRUE                       -0.310
fctr(ltrlty)l                     -0.306  0.504
fctr(ltrlty)r                     -0.306  0.504    0.500
fctr(ntrrt)                       -0.306  0.504    0.500         0.500
fctr(cngrty)TRUE:fctr(ltrlty)l     0.218 -0.706   -0.714        -0.357        -0.357
fctr(cngrty)TRUE:fctr(ltrlty)r     0.218 -0.706   -0.357        -0.714        -0.357   0.500
fctr(cngrty)TRUE:fctr(n)           0.218 -0.706   -0.357        -0.357        -0.714   0.500                          0.500
fctr(ltrlty)l:()                   0.216 -0.357   -0.707        -0.354        -0.707   0.505                          0.252
fctr(ltrlty)r:()                   0.216 -0.357   -0.354        -0.707        -0.707   0.252                          0.505
fctr(cngrty)TRUE:fctr(ltrlty)l:() -0.154  0.499    0.505         0.252         0.505  -0.707                         -0.354
fctr(cngrty)TRUE:fctr(ltrlty)r:() -0.154  0.499    0.252         0.505         0.505  -0.354                         -0.707
fctr(cngrty)TRUE:fctr(n) fctr(ltrlty)l:() fctr(ltrlty)r:() fctr(cngrty)TRUE:fctr(ltrlty)l:()
fctr(c)TRUE
fctr(ltrlty)l
fctr(ltrlty)r
fctr(ntrrt)
fctr(cngrty)TRUE:fctr(ltrlty)l
fctr(cngrty)TRUE:fctr(ltrlty)r
fctr(cngrty)TRUE:fctr(n)
fctr(ltrlty)l:()                   0.505
fctr(ltrlty)r:()                   0.505                    0.500
fctr(cngrty)TRUE:fctr(ltrlty)l:() -0.707                   -0.714           -0.357
fctr(cngrty)TRUE:fctr(ltrlty)r:() -0.707                   -0.357           -0.714            0.500
> anova(final.mod)
Analysis of Variance Table of type III  with  Satterthwaite
approximation for degrees of freedom
Sum Sq Mean Sq NumDF DenDF F.value    Pr(>F)
factor(congruity)                                        4439.1  4439.1     1 46142  39.850 2.768e-10 ***
factor(laterality)                                        572.9   286.5     2 46095   2.572  0.076430 .
factor(anteriority)                                      1508.1  1508.1     1 46095  13.538  0.000234 ***
factor(congruity):factor(laterality)                       31.6    15.8     2 46095   0.142  0.867581
factor(congruity):factor(anteriority)                     775.1   775.1     1 46095   6.958  0.008349 **
factor(laterality):factor(anteriority)                    111.9    56.0     2 46095   0.502  0.605126
factor(congruity):factor(laterality):factor(anteriority)   31.2    15.6     2 46095   0.140  0.869183
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


> congruity.mod<-lmer(uV~1+factor(congruity)+(1|sent.id)+(1|Subject),data=selected.data)
> summary(congruity.mod)
Linear mixed model fit by REML
t-tests use  Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: uV ~ 1 + factor(congruity) + (1 | sent.id) + (1 | Subject)
Data: selected.data
REML criterion at convergence: 494077.2
Scaled residuals:
Min       1Q   Median       3Q      Max
-10.1673  -0.5790  -0.0097   0.5818  12.6088

Random effects:
Groups   Name        Variance Std.Dev.
sent.id  (Intercept)   4.568   2.137
Subject  (Intercept)   6.132   2.476
Residual             178.137  13.347
Number of obs: 61568, groups:  sent.id, 41; Subject, 30

Fixed effects:
Estimate Std. Error         df t value Pr(>|t|)
(Intercept)                0.6055     0.5671    57.0000   1.068     0.29
factor(congruity)FALSE    -0.7105     0.1084 61535.0000  -6.558 5.51e-11 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
(Intr)
fctr()FALSE -0.093
> anova(congruity.mod)
Analysis of Variance Table of type III  with  Satterthwaite
approximation for degrees of freedom
Sum Sq Mean Sq NumDF DenDF F.value    Pr(>F)
factor(congruity) 7660.5  7660.5     1 61535  43.004 5.507e-11 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> laterality.mod<-lmer(uV~1+factor(laterality)+(1|sent.id)+(1|Subject),data=selected.data)
> summary(laterality.mod)
Linear mixed model fit by REML
t-tests use  Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: uV ~ 1 + factor(laterality) + (1 | sent.id) + (1 | Subject)
Data: selected.data

REML criterion at convergence: 372848.2

Scaled residuals:
Min      1Q  Median      3Q     Max
-9.7033 -0.5981 -0.0076  0.6006 12.2265

Random effects:
Groups   Name        Variance Std.Dev.
sent.id  (Intercept)   5.568   2.360
Subject  (Intercept)   6.777   2.603
Residual             186.966  13.674
Number of obs: 46176, groups:  sent.id, 41; Subject, 30

Fixed effects:
Estimate Std. Error         df t value Pr(>|t|)
(Intercept)                 0.8128     0.6115    61.0000   1.329  0.18877
factor(laterality)left     -0.4260     0.1559 46105.0000  -2.733  0.00628 **
factor(laterality)right    -0.6709     0.1559 46105.0000  -4.304 1.68e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
(Intr) fctr(ltrlty)l
fctr(ltrlty)l -0.127
fctr(ltrlty)r -0.127  0.500
> anova(laterality.mod)
Analysis of Variance Table of type III  with  Satterthwaite
approximation for degrees of freedom
Sum Sq Mean Sq NumDF DenDF F.value    Pr(>F)
factor(laterality) 3548.2  1774.1     2 46105  9.4889 7.584e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> anteriority.mod<-lmer(uV~1+factor(anteriority)+(1|sent.id)+(1|Subject),data=selected.data)
> summary(anteriority.mod)
Linear mixed model fit by REML
t-tests use  Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: uV ~ 1 + factor(anteriority) + (1 | sent.id) + (1 | Subject)
Data: selected.data

REML criterion at convergence: 372738.6

Scaled residuals:
Min      1Q  Median      3Q     Max
-9.6668 -0.5986 -0.0032  0.6017 12.2711

Random effects:
Groups   Name        Variance Std.Dev.
sent.id  (Intercept)   5.569   2.360
Subject  (Intercept)   6.777   2.603
Residual             186.525  13.657
Number of obs: 46176, groups:  sent.id, 41; Subject, 30

Fixed effects:
Estimate Std. Error         df t value Pr(>|t|)
(Intercept)                     -0.2693     0.6081    59.0000  -0.443     0.66
factor(anteriority)posterior     1.4328     0.1271 46105.0000  11.272   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
(Intr)
fctr(ntrrt) -0.105
> anova(anteriority.mod)
Analysis of Variance Table of type III  with  Satterthwaite
approximation for degrees of freedom
Sum Sq Mean Sq NumDF DenDF F.value    Pr(>F)
factor(anteriority)  23700   23700     1 46106  127.06 < 2.2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


Update: After updating the contrasts based on @Henrik's answer:

> options(contrasts=c("contr.sum","contr.poly"))
> final.mod<-lmer(uV~1+factor(congruity)*factor(laterality)*factor(anteriority)+(1|sent.id)+(1|Subject),data=selected.data)
> summary(final.mod)
Linear mixed model fit by REML
t-tests use  Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: uV ~ 1 + factor(congruity) * factor(laterality) *     factor(anteriority) +      (1 | sent.id) + (1 | Subject)
Data: selected.data

REML criterion at convergence: 372689.8

Scaled residuals:
Min      1Q  Median      3Q     Max
-9.6772 -0.5979 -0.0016  0.5977 12.3439

Random effects:
Groups   Name        Variance Std.Dev.
sent.id  (Intercept)   5.556   2.357
Subject  (Intercept)   6.752   2.599
Residual             186.232  13.647
Number of obs: 46176, groups:  sent.id, 41; Subject, 30

Fixed effects:
Estimate Std. Error         df t value Pr(>|t|)
(Intercept)                                                  4.355e-01  6.039e-01  5.800e+01   0.721   0.4737
factor(congruity)1                                           4.501e-01  6.396e-02  4.613e+04   7.037 1.99e-12 ***
factor(laterality)1                                          3.628e-01  8.983e-02  4.610e+04   4.039 5.38e-05 ***
factor(laterality)2                                         -5.732e-02  8.983e-02  4.610e+04  -0.638   0.5234
factor(anteriority)1                                        -7.183e-01  6.352e-02  4.610e+04 -11.308  < 2e-16 ***
factor(congruity)1:factor(laterality)1                       1.433e-01  8.983e-02  4.610e+04   1.596   0.1106
factor(congruity)1:factor(laterality)2                      -1.535e-01  8.983e-02  4.610e+04  -1.709   0.0875 .
factor(congruity)1:factor(anteriority)1                      9.442e-02  6.352e-02  4.610e+04   1.487   0.1371
factor(laterality)1:factor(anteriority)1                     2.282e-01  8.983e-02  4.610e+04   2.540   0.0111 *
factor(laterality)2:factor(anteriority)1                    -2.121e-01  8.983e-02  4.610e+04  -2.362   0.0182 *
factor(congruity)1:factor(laterality)1:factor(anteriority)1 -7.802e-03  8.983e-02  4.610e+04  -0.087   0.9308
factor(congruity)1:factor(laterality)2:factor(anteriority)1 -1.141e-02  8.983e-02  4.610e+04  -0.127   0.8989
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
(Intr) fctr(c)1 fctr(l)1 fct()2 fctr(n)1     fctr(cngrty)1:fctr(l)1 fc()1:()2 fctr(cngrty)1:fctr(n)1
fctr(cngr)1            -0.003
fctr(ltrl)1             0.000  0.000
fctr(ltrl)2             0.000  0.000   -0.500
fctr(ntrr)1             0.000  0.000    0.000    0.000
fctr(cngrty)1:fctr(l)1  0.000  0.000   -0.020    0.010  0.000
fctr()1:()2             0.000  0.000    0.010   -0.020  0.000   -0.500
fctr(cngrty)1:fctr(n)1  0.000  0.000    0.000    0.000 -0.020    0.000                  0.000
fctr(l)1:()1            0.000  0.000    0.000    0.000  0.000    0.000                  0.000     0.000
fctr()2:()1             0.000  0.000    0.000    0.000  0.000    0.000                  0.000     0.000
f()1:()1:()             0.000  0.000    0.000    0.000  0.000    0.000                  0.000     0.000
f()1:()2:()             0.000  0.000    0.000    0.000  0.000    0.000                  0.000     0.000
fctr(l)1:()1 f()2:( f()1:()1:
fctr(cngr)1
fctr(ltrl)1
fctr(ltrl)2
fctr(ntrr)1
fctr(cngrty)1:fctr(l)1
fctr()1:()2
fctr(cngrty)1:fctr(n)1
fctr(l)1:()1
fctr()2:()1            -0.500
f()1:()1:()            -0.020        0.010
f()1:()2:()             0.010       -0.020 -0.500
> anova(final.mod)
Analysis of Variance Table of type III  with  Satterthwaite
approximation for degrees of freedom
Sum Sq Mean Sq NumDF DenDF F.value    Pr(>F)
factor(congruity)                                         9221.9  9221.9     1 46129  49.518 1.993e-12 ***
factor(laterality)                                        3511.5  1755.7     2 46095   9.428 8.062e-05 ***
factor(anteriority)                                      23814.0 23814.0     1 46095 127.873 < 2.2e-16 ***
factor(congruity):factor(laterality)                       680.3   340.1     2 46095   1.826   0.16101
factor(congruity):factor(anteriority)                      411.5   411.5     1 46095   2.210   0.13714
factor(laterality):factor(anteriority)                    1497.4   748.7     2 46095   4.020   0.01796 *
factor(congruity):factor(laterality):factor(anteriority)     8.6     4.3     2 46095   0.023   0.97713
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

• Is this is a balanced design? Also, I assume that sent.id is the sensor id? If so, it appears that you have a random effect for the sensor locations and fixed effects for the sensor locations. Commented May 12, 2017 at 11:05
• The design is in principle balanced, though there are some missing data (~5% off the top of my head) which are however more or less evenly distributed over the cells. sent.id is sentence id - the stimuli are sentences and so there is a random effects for those. Commented May 13, 2017 at 18:36
• +1. Take a look at some of the top results in this search: stats.stackexchange.com/search?q=%5Blme4-nlme%5D+anova+summary - there might be something relevant. Are both anova() and summary() from lmerMod? Commented May 15, 2017 at 9:18
• In particular, see stats.stackexchange.com/a/265029/28666. Commented May 15, 2017 at 9:23
• Commented May 16, 2017 at 15:44

Type III tests require correct coding for lower-order effects to be meaningful, specifically orthogonal contrasts. The R default contr.treatment is not orthogonal, other contrasts are though (e.g., contr.sum). In your code it looks like you used did not change the defaults, hence your results are so-called simple main effects. We discuss this in our soon-to-come out chapter here, but other references are easy to find.

To use the correct contrasts run the following before fitting a mixed model in R:

options(contrasts=c("contr.sum","contr.poly"))


An easier to remember code is to use set_sum_contrasts() from my afex package:

afex::set_sum_contrasts()


Please update your question if this does not resolve your problem (preferably with data to recreate the problem).

• (By the way, I am wondering if this Q should be closed as a duplicate of stats.stackexchange.com/questions/249884 after you collect your bounty and your answer is hopefully accepted. Perhaps you will want to post an answer in that thread too.) Commented May 16, 2017 at 15:07
• I've edited the post to show the results with the updated contrasts. As you can see, that does eliminate the discrepancy. I've not had time to read your references yet, but I hope that will also help me to understand exactly how the difference in contrasts creates such a major difference here. I agree that this question is essentially a duplicate of the aforementioned post (I did search before, but without the lme4-nlme tag I got too many unrelated answers to find anything useful), but indeed I do think it is helpful to put the chapter and/or other references there. Commented May 16, 2017 at 15:34