2
$\begingroup$

I'm sure there is an obvious answer to this question but I'd like to understand better what the difference is between p-values for t-tests and type 3 ANOVA f-tests for binary variables in mixed effect models as implemented in lmerTest. In a standard linear model, my understanding is that these should be the same. For instance, using the ham dataset included in lmerTest:

library(lme4)
library(lmerTest)
library(car)

fm1 <- lm(Informed.liking ~ Gender + Information * Product, data=ham)

The p-values for Gender and Information in these models are the same, as expected (output cut to just the relevant parts).

summary(fm1)

Coefficients:
                      Estimate Std. Error t value Pr(>|t|)    
Gender2                -0.2443     0.1791  -1.364   0.1731    
Information2            0.1605     0.3582   0.448   0.6543 

Anova(fm1, type = 3)

Anova Table (Type III tests)

Response: Informed.liking
                    Sum Sq  Df  F value    Pr(>F)    
Gender                 9.7   1   1.8601  0.173090    
Information            1.0   1   0.2008  0.654260

However, running the same thing with a mixed effect model yields different p-values.

fm2 <- lmer(Informed.liking ~ Gender + Information * Product + (1 | Consumer), data=ham)

With the result

summary(fm2)

Fixed effects:
                      Estimate Std. Error       df t value Pr(>|t|)    
Gender2                -0.2443     0.2606  79.0000  -0.938   0.3514    
Information2            0.1605     0.3288 560.0000   0.488   0.6256   

anova(fm2, type = 3)

Type III Analysis of Variance Table with Satterthwaite's method
                    Sum Sq Mean Sq NumDF DenDF F value    Pr(>F)    
Gender               3.848  3.8480     1    79  0.8789 0.3513501    
Information          6.520  6.5201     1   560  1.4893 0.2228402

My understanding is that these are using the same Satterthwaite approximation. Does that not imply that these p-values are the same?

Edit: Thanks to Sal Magnifico below for pointing out the need for Sum contrast coding in car. This seems to solve the problem -- presumably lmerTest somehow does this automatically when calling a type 3 ANOVA. With contrast coding, the summary now matches the ANOVA for variables with 1 df. Here is the full reprex output including the Product terms:

library(reprex)
library(lme4)
#> Warning: package 'lme4' was built under R version 4.1.2
#> Loading required package: Matrix
library(lmerTest)
#> 
#> Attaching package: 'lmerTest'
#> The following object is masked from 'package:lme4':
#> 
#>     lmer
#> The following object is masked from 'package:stats':
#> 
#>     step
library(car)
#> Loading required package: carData
fm1 <- lm(Informed.liking ~ Gender + Information * Product, data=ham,
          contrasts=list(Gender=contr.sum, Information=contr.sum, Product=contr.sum))
summary(fm1)
#> 
#> Call:
#> lm(formula = Informed.liking ~ Gender + Information * Product, 
#>     data = ham, contrasts = list(Gender = contr.sum, Information = contr.sum, 
#>         Product = contr.sum))
#> 
#> Residuals:
#>     Min      1Q  Median      3Q     Max 
#> -5.4293 -1.7035  0.2471  1.8150  4.2224 
#> 
#> Coefficients:
#>                       Estimate Std. Error t value Pr(>|t|)    
#> (Intercept)            5.73152    0.08956  64.000  < 2e-16 ***
#> Gender1                0.12214    0.08956   1.364   0.1731    
#> Information1          -0.10031    0.08955  -1.120   0.2631    
#> Product1               0.07562    0.15510   0.488   0.6261    
#> Product2              -0.62809    0.15510  -4.049 5.76e-05 ***
#> Product3               0.35957    0.15510   2.318   0.0208 *  
#> Information1:Product1  0.02006    0.15510   0.129   0.8971    
#> Information1:Product2 -0.10340    0.15510  -0.667   0.5053    
#> Information1:Product3 -0.11574    0.15510  -0.746   0.4558    
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Residual standard error: 2.28 on 639 degrees of freedom
#> Multiple R-squared:  0.03442,    Adjusted R-squared:  0.02234 
#> F-statistic: 2.848 on 8 and 639 DF,  p-value: 0.004085
Anova(fm1, type = 3)
#> Anova Table (Type III tests)
#> 
#> Response: Informed.liking
#>                      Sum Sq  Df   F value    Pr(>F)    
#> (Intercept)         21283.7   1 4095.9447 < 2.2e-16 ***
#> Gender                  9.7   1    1.8601 0.1730900    
#> Information             6.5   1    1.2548 0.2630677    
#> Product                91.8   3    5.8893 0.0005733 ***
#> Information:Product    10.4   3    0.6663 0.5729402    
#> Residuals            3320.4 639                        
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

fm2 <- lmer(Informed.liking ~ Gender + Information * Product + (1 | Consumer), data=ham)
summary(fm2)
#> Linear mixed model fit by REML. t-tests use Satterthwaite's method [
#> lmerModLmerTest]
#> Formula: Informed.liking ~ Gender + Information * Product + (1 | Consumer)
#>    Data: ham
#> 
#> REML criterion at convergence: 2869.9
#> 
#> Scaled residuals: 
#>      Min       1Q   Median       3Q      Max 
#> -2.49290 -0.69971  0.09928  0.74897  2.69673 
#> 
#> Random effects:
#>  Groups   Name        Variance Std.Dev.
#>  Consumer (Intercept) 0.8274   0.9096  
#>  Residual             4.3780   2.0924  
#> Number of obs: 648, groups:  Consumer, 81
#> 
#> Fixed effects:
#>                       Estimate Std. Error       df t value Pr(>|t|)    
#> (Intercept)             5.8490     0.2843 358.4421  20.574   <2e-16 ***
#> Gender2                -0.2443     0.2606  79.0000  -0.938   0.3514    
#> Information2            0.1605     0.3288 560.0000   0.488   0.6256    
#> Product2               -0.8272     0.3288 560.0000  -2.516   0.0122 *  
#> Product3                0.1481     0.3288 560.0000   0.451   0.6525    
#> Product4                0.2963     0.3288 560.0000   0.901   0.3679    
#> Information2:Product2   0.2469     0.4650 560.0000   0.531   0.5956    
#> Information2:Product3   0.2716     0.4650 560.0000   0.584   0.5594    
#> Information2:Product4  -0.3580     0.4650 560.0000  -0.770   0.4416    
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Correlation of Fixed Effects:
#>             (Intr) Gendr2 Infrm2 Prdct2 Prdct3 Prdct4 In2:P2 In2:P3
#> Gender2     -0.453                                                 
#> Informatin2 -0.578  0.000                                          
#> Product2    -0.578  0.000  0.500                                   
#> Product3    -0.578  0.000  0.500  0.500                            
#> Product4    -0.578  0.000  0.500  0.500  0.500                     
#> Infrmtn2:P2  0.409  0.000 -0.707 -0.707 -0.354 -0.354              
#> Infrmtn2:P3  0.409  0.000 -0.707 -0.354 -0.707 -0.354  0.500       
#> Infrmtn2:P4  0.409  0.000 -0.707 -0.354 -0.354 -0.707  0.500  0.500
anova(fm2, type = 3)
#> Type III Analysis of Variance Table with Satterthwaite's method
#>                     Sum Sq Mean Sq NumDF DenDF F value    Pr(>F)    
#> Gender               3.848  3.8480     1    79  0.8789 0.3513501    
#> Information          6.520  6.5201     1   560  1.4893 0.2228402    
#> Product             91.807 30.6024     3   560  6.9901 0.0001271 ***
#> Information:Product 10.387  3.4624     3   560  0.7909 0.4992920    
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

fm3 <- lmer(Informed.liking ~ Gender + Information * Product + (1 | Consumer), data=ham,
            contrasts=list(Gender=contr.sum, Information=contr.sum, Product=contr.sum))
summary(fm3)
#> Linear mixed model fit by REML. t-tests use Satterthwaite's method [
#> lmerModLmerTest]
#> Formula: Informed.liking ~ Gender + Information * Product + (1 | Consumer)
#>    Data: ham
#> 
#> REML criterion at convergence: 2882.4
#> 
#> Scaled residuals: 
#>      Min       1Q   Median       3Q      Max 
#> -2.49290 -0.69971  0.09928  0.74897  2.69673 
#> 
#> Random effects:
#>  Groups   Name        Variance Std.Dev.
#>  Consumer (Intercept) 0.8274   0.9096  
#>  Residual             4.3780   2.0924  
#> Number of obs: 648, groups:  Consumer, 81
#> 
#> Fixed effects:
#>                        Estimate Std. Error        df t value Pr(>|t|)    
#> (Intercept)             5.73152    0.13028  79.00000  43.993  < 2e-16 ***
#> Gender1                 0.12214    0.13028  79.00000   0.938   0.3514    
#> Information1           -0.10031    0.08220 560.00000  -1.220   0.2228    
#> Product1                0.07562    0.14237 560.00000   0.531   0.5955    
#> Product2               -0.62809    0.14237 560.00000  -4.412 1.23e-05 ***
#> Product3                0.35957    0.14237 560.00000   2.526   0.0118 *  
#> Information1:Product1   0.02006    0.14237 560.00000   0.141   0.8880    
#> Information1:Product2  -0.10340    0.14237 560.00000  -0.726   0.4680    
#> Information1:Product3  -0.11574    0.14237 560.00000  -0.813   0.4166    
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Correlation of Fixed Effects:
#>             (Intr) Gendr1 Infrm1 Prdct1 Prdct2 Prdct3 In1:P1 In1:P2
#> Gender1     -0.012                                                 
#> Informatin1  0.000  0.000                                          
#> Product1     0.000  0.000  0.000                                   
#> Product2     0.000  0.000  0.000 -0.333                            
#> Product3     0.000  0.000  0.000 -0.333 -0.333                     
#> Infrmtn1:P1  0.000  0.000  0.000  0.000  0.000  0.000              
#> Infrmtn1:P2  0.000  0.000  0.000  0.000  0.000  0.000 -0.333       
#> Infrmtn1:P3  0.000  0.000  0.000  0.000  0.000  0.000 -0.333 -0.333
anova(fm3, type = 3)
#> Type III Analysis of Variance Table with Satterthwaite's method
#>                     Sum Sq Mean Sq NumDF DenDF F value    Pr(>F)    
#> Gender               3.848  3.8480     1    79  0.8789 0.3513501    
#> Information          6.520  6.5201     1   560  1.4893 0.2228402    
#> Product             91.807 30.6024     3   560  6.9901 0.0001271 ***
#> Information:Product 10.387  3.4624     3   560  0.7909 0.4992920    
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Created on 2023-04-17 by the reprex package (v2.0.1)

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ What happened to Product? $\endgroup$
    – Christian Hennig
    Commented Apr 17, 2023 at 10:39
  • $\begingroup$ One thing is that at least with OLS lm() models in R, you need to change the contrast coding in the model to get type III sums of squares, like, model.3 = lm(Y ~ A + B + A:B, contrasts=list(A="contr.sum", B="contr.sum")) . $\endgroup$
    – Sal Mangiafico
    Commented Apr 17, 2023 at 12:15
  • $\begingroup$ Note that this is included the documentation for car::Anova in the examples. www.rdocumentation.org/packages/car/versions/1.0-9/topics/Anova . $\endgroup$
    – Sal Mangiafico
    Commented Apr 17, 2023 at 12:17
  • 1
    $\begingroup$ Thanks so much to both of you! Sal Mangiafico seems to have identified the issue, it seems like lmerTest must switch to sum coding under the hood when you ask for a type 3 anova. After switching, summary now matches the anova output for variables with 1 df, as expected -- see the edit above which includes the full table including the Product terms for Christian Hennig as well. $\endgroup$
    – dhalpern
    Commented Apr 17, 2023 at 12:37
  • 1
    $\begingroup$ @SalMangiafico Your comment seems very close to an answer. I'm trying to reduce the number of unanswered questions here. Do yo want to make your comment into an answer? $\endgroup$
    – Peter Flom
    Commented Dec 14, 2023 at 11:04

1 Answer 1

Reset to default
1
$\begingroup$

This is largely a software question, and answered in the question itself and in the comments.

Answer from comments:

With OLS lm() models in R, you need to change the contrast coding in the model to get type III sums of squares, like, model.3 = lm(Y ~ A + B + A:B, contrasts=list(A="contr.sum", B="contr.sum"))

Note that this is included the documentation for car::Anova in the examples. www.rdocumentation.org/packages/car/versions/1.0-9/topics/Anova.

In the edit, OP reports, "This seems to solve the problem -- presumably lmerTest somehow does this automatically when calling a type 3 ANOVA. With contrast coding, the summary now matches the ANOVA for variables with 1 df."

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.