# Why are results different when using aov_ez{afex} and Anova{car}, Type III SS in R?

Hopefully, the answer to this question is simple.

Why do I get different results when I am using Anova from the car package and the aov_ez from the afex package?

library(car)
library(afex)

set.seed(100)

dv <-  runif(30,90,110)
factor_a <-  factor(rep(c("a","b"),15))
factor_b <-  factor(rep(c("f","m"),each=15))
id <- factor(1:30)
data <- data.frame(dv,factor_a,factor_b,id)

data$dv[data$factor_b=="f" & data$factor_a=="a"] <- data$dv[data$factor_b=="f" & data$factor_a=="a"] *1.05

aov_ez(dv="dv",between = c("factor_a","factor_b"), id="id",data=data )
Anova(lm(dv~factor_a*factor_b, data=data),type = "III")


Results are different in case of the main effect factor B: Using aov_ez, the main effect is significant, using Anova, the main effect is not significant. How is that possible? In both cases, there is a significant interaction and the F-values are very similar. But the F-values of the main effects differ.

>     aov_ez(dv="dv",between = c("factor_a","factor_b"), id="id",data=data )
Contrasts set to contr.sum for the following variables: factor_a, factor_b
Anova Table (Type 3 tests)

Response: dv
Effect    df   MSE       F ges p.value
1          factor_a 1, 26 20.96  4.16 + .14     .05
2          factor_b 1, 26 20.96    0.55 .02     .46
3 factor_a:factor_b 1, 26 20.96 9.03 ** .26    .006
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘+’ 0.1 ‘ ’ 1

>     Anova(lm(dv~factor_a*factor_b, data=data),type = "III")
Anova Table (Type III tests)

Response: dv
Sum Sq Df   F value    Pr(>F)
(Intercept)        89775  1 4282.2927 < 2.2e-16 ***
factor_a             267  1   12.7159  0.001434 **
factor_b             147  1    7.0288  0.013478 *
factor_a:factor_b    189  1    9.0269  0.005822 **
Residuals            545 26
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


Because I am quite new using R, which of these two packages I should use. Does anyone know why the results differ?

• Shouldn't data$factor_a = "a" be data$factor_a == "a"? Commented Aug 11, 2017 at 18:25
• oh thank you! i didn't noticed that earlier. do you also have any idea why these two functions differ with respect to the results? Commented Aug 11, 2017 at 20:32
• Did you try running the corrected code? Also, set the RNG seed so your results are reproducible. Commented Aug 12, 2017 at 1:21
• hey kodiologist and thank you again for you comment. I did run it again with the corrected code and also set seed to 100. The outcome is especially interesting because now, using the car package, I have 2 significant main effects as well as a significant interaction (the only value that is identical with the output of the afex package function). MSE = 20.96 is also equal to 545/26 = 20.96, this can't be it Commented Aug 12, 2017 at 15:44

First note that you have a balanced between-subjects design with only dichotomous IVs, so the sum-of-squares type doesn't matter and Anova(lm(dv~factor_a*factor_b, data=data)) produces the same $p$-values as $t$-tests of regression coefficients:

> print(summary(lm(dv~factor_a*factor_b, data=data)))
[…]
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)          105.933      1.619  65.439  < 2e-16 ***
factor_ab             -8.450      2.370  -3.566  0.00143 **
factor_bm             -6.282      2.370  -2.651  0.01348 *
factor_ab:factor_bm   10.069      3.351   3.004  0.00582 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
[…]


So why does aov_ez give different $p$s? It seems that aov_ez codes the IVs with effects coding by default, rather than dummy coding as in base R's lm; the message Contrasts set to contr.sum for the following variables: factor_a, factor_b is a warning of this. When we set check.contrasts to FALSE, aov_ez uses dummy coding and we get concordant $F$s and $p$s:

> print(aov_ez(dv="dv",between = c("factor_a","factor_b"), id="id",data=data, check_contrasts = F ))
Anova Table (Type 3 tests)

Response: dv
Effect    df   MSE        F ges p.value
1          factor_a 1, 26 20.96 12.72 ** .33    .001
2          factor_b 1, 26 20.96   7.03 * .21     .01
3 factor_a:factor_b 1, 26 20.96  9.03 ** .26    .006
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘+’ 0.1 ‘ ’ 1
Warning message:
Calculating Type 3 sums with contrasts != 'contr.sum' for: factor_a, factor_b, factor_a, factor_b
Results likely bogus or not interpretable!
You probably want check_contrasts = TRUE or options(contrasts=c('contr.sum','contr.poly'))


Why exactly the package author thinks that dummy coding in this situation makes the results "likely bogus or not interpretable" goes further into the weeds of ANOVA than I'm familiar with. But at least now we know what aov_ez is doing.