7
$\begingroup$

While helping someone else with their analyses, I've run into a question regarding the difference between t-tests and F-tests for linear mixed models in lme4 for R, as provided by lmerTest. I'm aware of the problems with calculating any kind of p-values for linear mixed models (as I understand, primarily due to the fact that definition of the degrees of freedom is problematic), as well as the problems with interpreting main effects in the presence of significant interactions (based on the marginality principle).

Briefly, the data are from an experiment with two conditions (congruity TRUE/FALSE), measured on six sets of sensors which can be described as a combination of two factors: anteriority (anterior/posterior) and laterality (left/central/right).

As can be seen from the summary output below, the t.tests do not show a significant congruity effect (p = 0.12), while the anova output shows a very significant congruity effect (p = 2.8e-10). Since congruity has only two levels, this cannot be the result of the F-test doing an omnibus test over several levels of the fixed factor. I am therefore unsure what causes the very significant result in the anova output. Is this due to the fact that there are strong interactions involving congruity which of course depend on the inclusion of the main effect in the model parametrization?

I have looked for a previous answer to this question on CrossValidated but I have not been able to find anything relevant except possibly the first answer to this question. However, if that does provide a real answer then it is implicit in the mathematics, and I am looking for a conceptual answer that I can explain to the person I am trying to help.

> final.mod<-lmer(uV~1+factor(congruity)*factor(laterality)*factor(anteriority)+(1|sent.id)+(1|Subject),data=selected.data)
> summary(final.mod)
Linear mixed model fit by REML 

t-tests use  Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: uV ~ 1 + factor(congruity) * factor(laterality) * factor(anteriority) +      (1 | sent.id) + (1 | Subject)
   Data: selected.data
REML criterion at convergence: 348903.5
Scaled residuals: 
Min      1Q  Median      3Q     Max 
-7.0440 -0.6002  0.0069  0.6038 11.3912 
Random effects:
 Groups   Name        Variance Std.Dev.
 sent.id  (Intercept)   1.773   1.332  
 Subject  (Intercept)   2.548   1.596  
 Residual             111.396  10.554  
Number of obs: 46176, groups:  sent.id, 41; Subject, 30
Fixed effects:
                                                                             Estimate Std. Error         df t value Pr(>|t|)  
(Intercept)                                                                 4.768e-03  3.973e-01  7.900e+01   0.012   0.9905  
factor(congruity)TRUE                                                       3.758e-01  2.410e-01  4.611e+04   1.559   0.1189  
factor(laterality)left                                                      7.154e-02  2.430e-01  4.610e+04   0.294   0.7685  
factor(laterality)right                                                    -2.003e-01  2.430e-01  4.610e+04  -0.824   0.4098  
factor(anteriority)posterior                                               -4.203e-02  2.430e-01  4.610e+04  -0.173   0.8627
factor(congruity)TRUE:factor(laterality)left                               -1.013e-01  3.404e-01  4.610e+04  -0.298   0.7660
factor(congruity)TRUE:factor(laterality)right                               7.233e-02  3.404e-01  4.610e+04   0.213   0.8317
factor(congruity)TRUE:factor(anteriority)posterior                          6.162e-01  3.404e-01  4.610e+04   1.810   0.0702 .
factor(laterality)left:factor(anteriority)posterior                         2.568e-01  3.437e-01  4.610e+04   0.747   0.4549
factor(laterality)right:factor(anteriority)posterior                        1.763e-01  3.437e-01  4.610e+04   0.513   0.6080
factor(congruity)TRUE:factor(laterality)left:factor(anteriority)posterior  -5.162e-02  4.813e-01  4.610e+04  -0.107   0.9146
factor(congruity)TRUE:factor(laterality)right:factor(anteriority)posterior -2.420e-01  4.813e-01  4.610e+04  -0.503   0.6152  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Correlation of Fixed Effects:
                          (Intr) fc()TRUE fctr(ltrlty)l fctr(ltrlty)r fctr(n) fctr(cngrty)TRUE:fctr(ltrlty)l fctr(cngrty)TRUE:fctr(ltrlty)r
fctr(c)TRUE                       -0.310
fctr(ltrlty)l                     -0.306  0.504
fctr(ltrlty)r                     -0.306  0.504    0.500
fctr(ntrrt)                       -0.306  0.504    0.500         0.500
fctr(cngrty)TRUE:fctr(ltrlty)l     0.218 -0.706   -0.714        -0.357        -0.357
fctr(cngrty)TRUE:fctr(ltrlty)r     0.218 -0.706   -0.357        -0.714        -0.357   0.500
fctr(cngrty)TRUE:fctr(n)           0.218 -0.706   -0.357        -0.357        -0.714   0.500                          0.500
fctr(ltrlty)l:()                   0.216 -0.357   -0.707        -0.354        -0.707   0.505                          0.252
fctr(ltrlty)r:()                   0.216 -0.357   -0.354        -0.707        -0.707   0.252                          0.505
fctr(cngrty)TRUE:fctr(ltrlty)l:() -0.154  0.499    0.505         0.252         0.505  -0.707                         -0.354
fctr(cngrty)TRUE:fctr(ltrlty)r:() -0.154  0.499    0.252         0.505         0.505  -0.354                         -0.707                        
                          fctr(cngrty)TRUE:fctr(n) fctr(ltrlty)l:() fctr(ltrlty)r:() fctr(cngrty)TRUE:fctr(ltrlty)l:()
fctr(c)TRUE
fctr(ltrlty)l
fctr(ltrlty)r
fctr(ntrrt)
fctr(cngrty)TRUE:fctr(ltrlty)l
fctr(cngrty)TRUE:fctr(ltrlty)r
fctr(cngrty)TRUE:fctr(n)
fctr(ltrlty)l:()                   0.505
fctr(ltrlty)r:()                   0.505                    0.500
fctr(cngrty)TRUE:fctr(ltrlty)l:() -0.707                   -0.714           -0.357                                            
fctr(cngrty)TRUE:fctr(ltrlty)r:() -0.707                   -0.357           -0.714            0.500                           
> anova(final.mod)
Analysis of Variance Table of type III  with  Satterthwaite 
approximation for degrees of freedom
                                                         Sum Sq Mean Sq NumDF DenDF F.value    Pr(>F)    
factor(congruity)                                        4439.1  4439.1     1 46142  39.850 2.768e-10 ***
factor(laterality)                                        572.9   286.5     2 46095   2.572  0.076430 .  
factor(anteriority)                                      1508.1  1508.1     1 46095  13.538  0.000234 ***
factor(congruity):factor(laterality)                       31.6    15.8     2 46095   0.142  0.867581    
factor(congruity):factor(anteriority)                     775.1   775.1     1 46095   6.958  0.008349 ** 
factor(laterality):factor(anteriority)                    111.9    56.0     2 46095   0.502  0.605126  
factor(congruity):factor(laterality):factor(anteriority)   31.2    15.6     2 46095   0.140  0.869183    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

In reply to @Aurelie's question:

> congruity.mod<-lmer(uV~1+factor(congruity)+(1|sent.id)+(1|Subject),data=selected.data)
> summary(congruity.mod)
Linear mixed model fit by REML 
t-tests use  Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: uV ~ 1 + factor(congruity) + (1 | sent.id) + (1 | Subject)
   Data: selected.data
REML criterion at convergence: 494077.2
Scaled residuals: 
     Min       1Q   Median       3Q      Max 
-10.1673  -0.5790  -0.0097   0.5818  12.6088 

Random effects:
 Groups   Name        Variance Std.Dev.
 sent.id  (Intercept)   4.568   2.137  
 Subject  (Intercept)   6.132   2.476  
 Residual             178.137  13.347  
Number of obs: 61568, groups:  sent.id, 41; Subject, 30

Fixed effects:
                         Estimate Std. Error         df t value Pr(>|t|)    
(Intercept)                0.6055     0.5671    57.0000   1.068     0.29    
factor(congruity)FALSE    -0.7105     0.1084 61535.0000  -6.558 5.51e-11 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
            (Intr)
fctr()FALSE -0.093
> anova(congruity.mod)
Analysis of Variance Table of type III  with  Satterthwaite 
approximation for degrees of freedom
                  Sum Sq Mean Sq NumDF DenDF F.value    Pr(>F)    
factor(congruity) 7660.5  7660.5     1 61535  43.004 5.507e-11 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> laterality.mod<-lmer(uV~1+factor(laterality)+(1|sent.id)+(1|Subject),data=selected.data)
> summary(laterality.mod)
Linear mixed model fit by REML 
t-tests use  Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: uV ~ 1 + factor(laterality) + (1 | sent.id) + (1 | Subject)
   Data: selected.data

REML criterion at convergence: 372848.2

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-9.7033 -0.5981 -0.0076  0.6006 12.2265 

Random effects:
 Groups   Name        Variance Std.Dev.
 sent.id  (Intercept)   5.568   2.360  
 Subject  (Intercept)   6.777   2.603  
 Residual             186.966  13.674  
Number of obs: 46176, groups:  sent.id, 41; Subject, 30

Fixed effects:
                          Estimate Std. Error         df t value Pr(>|t|)    
(Intercept)                 0.8128     0.6115    61.0000   1.329  0.18877    
factor(laterality)left     -0.4260     0.1559 46105.0000  -2.733  0.00628 ** 
factor(laterality)right    -0.6709     0.1559 46105.0000  -4.304 1.68e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
              (Intr) fctr(ltrlty)l
fctr(ltrlty)l -0.127              
fctr(ltrlty)r -0.127  0.500       
> anova(laterality.mod)
Analysis of Variance Table of type III  with  Satterthwaite 
approximation for degrees of freedom
                   Sum Sq Mean Sq NumDF DenDF F.value    Pr(>F)    
factor(laterality) 3548.2  1774.1     2 46105  9.4889 7.584e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> anteriority.mod<-lmer(uV~1+factor(anteriority)+(1|sent.id)+(1|Subject),data=selected.data)
> summary(anteriority.mod)
Linear mixed model fit by REML 
t-tests use  Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: uV ~ 1 + factor(anteriority) + (1 | sent.id) + (1 | Subject)
   Data: selected.data

REML criterion at convergence: 372738.6

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-9.6668 -0.5986 -0.0032  0.6017 12.2711 

Random effects:
 Groups   Name        Variance Std.Dev.
 sent.id  (Intercept)   5.569   2.360  
 Subject  (Intercept)   6.777   2.603  
 Residual             186.525  13.657  
Number of obs: 46176, groups:  sent.id, 41; Subject, 30

Fixed effects:
                           Estimate Std. Error         df t value Pr(>|t|)    
(Intercept)                     -0.2693     0.6081    59.0000  -0.443     0.66    
factor(anteriority)posterior     1.4328     0.1271 46105.0000  11.272   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
            (Intr)
fctr(ntrrt) -0.105
> anova(anteriority.mod)
Analysis of Variance Table of type III  with  Satterthwaite 
approximation for degrees of freedom
                    Sum Sq Mean Sq NumDF DenDF F.value    Pr(>F)    
factor(anteriority)  23700   23700     1 46106  127.06 < 2.2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Update: After updating the contrasts based on @Henrik's answer:

> options(contrasts=c("contr.sum","contr.poly"))
> final.mod<-lmer(uV~1+factor(congruity)*factor(laterality)*factor(anteriority)+(1|sent.id)+(1|Subject),data=selected.data)
> summary(final.mod)
Linear mixed model fit by REML 
t-tests use  Satterthwaite approximations to degrees of freedom ['lmerMod']
Formula: uV ~ 1 + factor(congruity) * factor(laterality) *     factor(anteriority) +      (1 | sent.id) + (1 | Subject)
   Data: selected.data

REML criterion at convergence: 372689.8

Scaled residuals: 
    Min      1Q  Median      3Q     Max 
-9.6772 -0.5979 -0.0016  0.5977 12.3439 

Random effects:
 Groups   Name        Variance Std.Dev.
 sent.id  (Intercept)   5.556   2.357  
 Subject  (Intercept)   6.752   2.599  
 Residual             186.232  13.647  
Number of obs: 46176, groups:  sent.id, 41; Subject, 30

Fixed effects:
                                                              Estimate Std. Error         df t value Pr(>|t|)    
(Intercept)                                                  4.355e-01  6.039e-01  5.800e+01   0.721   0.4737    
factor(congruity)1                                           4.501e-01  6.396e-02  4.613e+04   7.037 1.99e-12 ***
factor(laterality)1                                          3.628e-01  8.983e-02  4.610e+04   4.039 5.38e-05 ***
factor(laterality)2                                         -5.732e-02  8.983e-02  4.610e+04  -0.638   0.5234    
factor(anteriority)1                                        -7.183e-01  6.352e-02  4.610e+04 -11.308  < 2e-16 ***
factor(congruity)1:factor(laterality)1                       1.433e-01  8.983e-02  4.610e+04   1.596   0.1106    
factor(congruity)1:factor(laterality)2                      -1.535e-01  8.983e-02  4.610e+04  -1.709   0.0875 .  
factor(congruity)1:factor(anteriority)1                      9.442e-02  6.352e-02  4.610e+04   1.487   0.1371    
factor(laterality)1:factor(anteriority)1                     2.282e-01  8.983e-02  4.610e+04   2.540   0.0111 *  
factor(laterality)2:factor(anteriority)1                    -2.121e-01  8.983e-02  4.610e+04  -2.362   0.0182 *  
factor(congruity)1:factor(laterality)1:factor(anteriority)1 -7.802e-03  8.983e-02  4.610e+04  -0.087   0.9308    
factor(congruity)1:factor(laterality)2:factor(anteriority)1 -1.141e-02  8.983e-02  4.610e+04  -0.127   0.8989    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Correlation of Fixed Effects:
                       (Intr) fctr(c)1 fctr(l)1 fct()2 fctr(n)1     fctr(cngrty)1:fctr(l)1 fc()1:()2 fctr(cngrty)1:fctr(n)1
fctr(cngr)1            -0.003                                                                                          
fctr(ltrl)1             0.000  0.000                                                                                   
fctr(ltrl)2             0.000  0.000   -0.500                                                                          
fctr(ntrr)1             0.000  0.000    0.000    0.000                                                                 
fctr(cngrty)1:fctr(l)1  0.000  0.000   -0.020    0.010  0.000                                                          
fctr()1:()2             0.000  0.000    0.010   -0.020  0.000   -0.500                                                 
fctr(cngrty)1:fctr(n)1  0.000  0.000    0.000    0.000 -0.020    0.000                  0.000                          
fctr(l)1:()1            0.000  0.000    0.000    0.000  0.000    0.000                  0.000     0.000                
fctr()2:()1             0.000  0.000    0.000    0.000  0.000    0.000                  0.000     0.000                
f()1:()1:()             0.000  0.000    0.000    0.000  0.000    0.000                  0.000     0.000                
f()1:()2:()             0.000  0.000    0.000    0.000  0.000    0.000                  0.000     0.000                
                       fctr(l)1:()1 f()2:( f()1:()1:
fctr(cngr)1                                         
fctr(ltrl)1                                         
fctr(ltrl)2                                         
fctr(ntrr)1                                         
fctr(cngrty)1:fctr(l)1                              
fctr()1:()2                                         
fctr(cngrty)1:fctr(n)1                              
fctr(l)1:()1                                        
fctr()2:()1            -0.500                       
f()1:()1:()            -0.020        0.010          
f()1:()2:()             0.010       -0.020 -0.500   
> anova(final.mod)
Analysis of Variance Table of type III  with  Satterthwaite 
approximation for degrees of freedom
                                                          Sum Sq Mean Sq NumDF DenDF F.value    Pr(>F)    
factor(congruity)                                         9221.9  9221.9     1 46129  49.518 1.993e-12 ***
factor(laterality)                                        3511.5  1755.7     2 46095   9.428 8.062e-05 ***
factor(anteriority)                                      23814.0 23814.0     1 46095 127.873 < 2.2e-16 ***
factor(congruity):factor(laterality)                       680.3   340.1     2 46095   1.826   0.16101    
factor(congruity):factor(anteriority)                      411.5   411.5     1 46095   2.210   0.13714    
factor(laterality):factor(anteriority)                    1497.4   748.7     2 46095   4.020   0.01796 *  
factor(congruity):factor(laterality):factor(anteriority)     8.6     4.3     2 46095   0.023   0.97713    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Improve this question
$\endgroup$
6
  • $\begingroup$ Is this is a balanced design? Also, I assume that sent.id is the sensor id? If so, it appears that you have a random effect for the sensor locations and fixed effects for the sensor locations. $\endgroup$
    – dbwilson
    Commented May 12, 2017 at 11:05
  • $\begingroup$ The design is in principle balanced, though there are some missing data (~5% off the top of my head) which are however more or less evenly distributed over the cells. sent.id is sentence id - the stimuli are sentences and so there is a random effects for those. $\endgroup$
    – Ishisht
    Commented May 13, 2017 at 18:36
  • 1
    $\begingroup$ +1. Take a look at some of the top results in this search: stats.stackexchange.com/search?q=%5Blme4-nlme%5D+anova+summary - there might be something relevant. Are both anova() and summary() from lmerMod? $\endgroup$
    – amoeba
    Commented May 15, 2017 at 9:18
  • $\begingroup$ In particular, see stats.stackexchange.com/a/265029/28666. $\endgroup$
    – amoeba
    Commented May 15, 2017 at 9:23
  • 5
    $\begingroup$ Possible duplicate of Conflicting results of summary() and anova() for a mixed model with interactions in lmer+lmerTest $\endgroup$
    – Henrik
    Commented May 16, 2017 at 15:44

1 Answer 1

Reset to default
4
+50
$\begingroup$

Type III tests require correct coding for lower-order effects to be meaningful, specifically orthogonal contrasts. The R default contr.treatment is not orthogonal, other contrasts are though (e.g., contr.sum). In your code it looks like you used did not change the defaults, hence your results are so-called simple main effects. We discuss this in our soon-to-come out chapter here, but other references are easy to find.

To use the correct contrasts run the following before fitting a mixed model in R:

options(contrasts=c("contr.sum","contr.poly"))

An easier to remember code is to use set_sum_contrasts() from my afex package:

afex::set_sum_contrasts()

Please update your question if this does not resolve your problem (preferably with data to recreate the problem).

Improve this answer
$\endgroup$
2
  • $\begingroup$ (By the way, I am wondering if this Q should be closed as a duplicate of stats.stackexchange.com/questions/249884 after you collect your bounty and your answer is hopefully accepted. Perhaps you will want to post an answer in that thread too.) $\endgroup$
    – amoeba
    Commented May 16, 2017 at 15:07
  • 1
    $\begingroup$ I've edited the post to show the results with the updated contrasts. As you can see, that does eliminate the discrepancy. I've not had time to read your references yet, but I hope that will also help me to understand exactly how the difference in contrasts creates such a major difference here. I agree that this question is essentially a duplicate of the aforementioned post (I did search before, but without the lme4-nlme tag I got too many unrelated answers to find anything useful), but indeed I do think it is helpful to put the chapter and/or other references there. $\endgroup$
    – Ishisht
    Commented May 16, 2017 at 15:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.