2
$\begingroup$

I ran into something I do not understand when conducting a repeated measures ANOVA.

Short description: I'm using a dataset included in the ez package. When I conduct a repeated measures ANOVA on the full dataset, the results of ezANOVA and aov() are equivalent. However, once I take only a subset (in this case: only reaction times for trials without error) the results with ezANOVA and aov() differ.

The longer story: The dataset contains a subject column (subnum), two within subject factors (cue, flank) and the dependent variable (rt). The dataset can be loaded via

library('ez')
data(ANT)
df = ANT
df$cue <- as.factor(df$cue)
df$flank <- as.factor(df$flank)
df$subnum <- as.factor(df$subnum)

subnum group block trial cue  flank  location   direction   rt  error
1   Treatment   1   1   None    Neutral     up  left    398.6773    0
1   Treatment   1   2   Center  Neutral     up  left    389.1822    0
1   Treatment   1   3   Double  Neutral     up  left    333.2186    0
1   Treatment   1   4   Spatial Neutral     up  left    419.7640    0
1   Treatment   1   5   None    Congruent   up  left    446.4754    0
1   Treatment   1   6   Center  Congruent   up  left    338.9766    0
1   Treatment   1   7   Double  Congruent   up  left    399.3715    0

Now when I perform a repeated measures ANOVA on the full dataset, and use both ezANOVA and aov(),the results are the same.

ezANOVA(
  data=df,
  dv=rt,
  wid=subnum,
  within = .(cue, flank),
)

$ANOVA
    	Effect	DFn	DFd	F	p	p<.05	ges
    2	cue 	3 	57 	540.862407 	7.988172e-42	* 	0.87793881
    3	flank 	2 	38 	1066.037656 	4.196305e-34	* 	0.91110583
    4	cue:flank 	6 	114 	4.357093 	5.356773e-04	* 	0.09416982
$`Mauchly's Test for Sphericity`
        Effect  W   p   p<.05
    2   cue     0.8431739   0.69690404  
    3   flank   0.7999302   0.13411237  
    4   cue:flank   0.1378186   0.03419366  *
$`Sphericity Corrections`
        Effect  GGe p[GG]   p[GG]<.05   HFe p[HF]   p[HF]<.05
    2   cue     0.9016877   6.126025e-38    *   1.0657965   7.988172e-42    *
    3   flank   0.8332849   8.590878e-29    *   0.9037852   4.869100e-31    *
    4   cue:flank   0.5956263   4.652864e-03    *   0.7506166   2.015937e-03    *

For aov():

mod123 <- aov(rt ~ (cue*flank) + Error(subnum/(cue*flank)), data = df)
summary(mod123)


Error: subnum
          Df Sum Sq Mean Sq F value Pr(>F)
Residuals 19  85489    4499               

Error: subnum:cue
          Df  Sum Sq Mean Sq F value Pr(>F)    
cue        3 5523668 1841223   540.9 <2e-16 ***
Residuals 57  194041    3404                   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Error: subnum:flank
          Df  Sum Sq Mean Sq F value Pr(>F)    
flank      2 7871119 3935559    1066 <2e-16 ***
Residuals 38  140287    3692                   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Error: subnum:cue:flank
           Df Sum Sq Mean Sq F value   Pr(>F)    
cue:flank   6  79837   13306   4.357 0.000536 ***
Residuals 114 348147    3054                     
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Error: Within
            Df   Sum Sq Mean Sq F value Pr(>F)
Residuals 5520 14422221    2613

So until now, everything works fine. But if I select only those observations where error equals zero, the results between both approaches differ:

ezANOVA(
  data=df[df$error==0,],
  dv=rt,
  wid=subnum,
  within = .(cue, flank),
)

$ANOVA
     Effect DFn DFd          F            p p<.05        ges
2       cue   3  57 477.564650 2.435084e-40     * 0.86387868
3     flank   2  38 958.640865 3.040261e-33     * 0.90297213
4 cue:flank   6 114   4.047785 1.026734e-03     * 0.08633287

$`Mauchly's Test for Sphericity`
     Effect         W          p p<.05
2       cue 0.8670854 0.77271988      
3     flank 0.9088146 0.42293876      
4 cue:flank 0.1506008 0.04917243     *

$`Sphericity Corrections`
     Effect       GGe        p[GG] p[GG]<.05      HFe        p[HF] p[HF]<.05
2       cue 0.9165014 3.647676e-37         * 1.086943 2.435084e-40         *
3     flank 0.9164345 1.182224e-30         * 1.009411 3.040261e-33         *
4 cue:flank 0.6261487 6.059761e-03         * 0.799682 2.641207e-03         *

mod123 <- aov(rt ~ (cue*flank) + Error(subnum/(cue*flank)), data = df[df$error==0,])
summary(mod123)


Error: subnum
          Df Sum Sq Mean Sq F value Pr(>F)  
cue        3  22044    7348   2.037  0.187  
flank      2  31873   15936   4.418  0.051 .
cue:flank  6  12677    2113   0.586  0.735  
Residuals  8  28858    3607                 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Error: subnum:cue
          Df  Sum Sq Mean Sq F value Pr(>F)    
cue        3 4818840 1606280 445.910 <2e-16 ***
flank      2    3512    1756   0.487  0.617    
cue:flank  6   24626    4104   1.139  0.354    
Residuals 49  176510    3602                   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Error: subnum:flank
          Df  Sum Sq Mean Sq F value Pr(>F)    
flank      2 7195298 3597649 936.928 <2e-16 ***
cue:flank  6   17408    2901   0.756   0.61    
Residuals 32  122875    3840                   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Error: subnum:cue:flank
           Df Sum Sq Mean Sq F value   Pr(>F)    
cue:flank   6  73202   12200   4.096 0.000928 ***
Residuals 114 339584    2979                     
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Error: Within
            Df   Sum Sq Mean Sq F value Pr(>F)
Residuals 4951 12871045    2600

What am I missing here? Many thanks in advance.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

The difference comes from the fact that selecting observations with error == 0 gives you an unbalanced design. ezANOVA will take cell means for you and pretend that you have a balanced design, but aov will not.

Let's balance the design ourselves and see that the outputs match:

> data <- aggregate(rt ~ subnum + cue + flank, ANT[ANT$error == 0, ], mean)
> ezANOVA(data, dv=rt, wid=subnum, within=.(cue, flank))
$ANOVA
     Effect DFn DFd          F            p p<.05        ges
2       cue   3  57 477.564650 2.435084e-40     * 0.86387868
3     flank   2  38 958.640865 3.040261e-33     * 0.90297213
4 cue:flank   6 114   4.047785 1.026734e-03     * 0.08633287

$`Mauchly's Test for Sphericity`
     Effect         W          p p<.05
2       cue 0.8670854 0.77271988      
3     flank 0.9088146 0.42293876      
4 cue:flank 0.1506008 0.04917243     *

$`Sphericity Corrections`
     Effect       GGe        p[GG] p[GG]<.05      HFe        p[HF] p[HF]<.05
2       cue 0.9165014 3.647676e-37         * 1.086943 2.435084e-40         *
3     flank 0.9164345 1.182224e-30         * 1.009411 3.040261e-33         *
4 cue:flank 0.6261487 6.059761e-03         * 0.799682 2.641207e-03         *
> summary(aov(rt ~ cue * flank + Error(subnum / (cue * flank)), data))

Error: subnum
          Df Sum Sq Mean Sq F value Pr(>F)
Residuals 19   4247   223.5               

Error: subnum:cue
          Df Sum Sq Mean Sq F value Pr(>F)    
cue        3 225486   75162   477.6 <2e-16 ***
Residuals 57   8971     157                   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Error: subnum:flank
          Df Sum Sq Mean Sq F value Pr(>F)    
flank      2 330651  165326   958.6 <2e-16 ***
Residuals 38   6553     172                   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Error: subnum:cue:flank
           Df Sum Sq Mean Sq F value  Pr(>F)   
cue:flank   6   3357   559.5   4.048 0.00103 **
Residuals 114  15759   138.2                   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks a lot - of course! That unties the knots in my head. $\endgroup$
    – adswa
    Commented Jul 12, 2018 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.