Different F-ratios for within subjects effects when using SPSS and R's aov

I've just compared the ANOVA tables generated by SPSS and Statistica with the aov table provided by summary(aov.model). They yield identical between Subject effects (e.g., NativeLanguage(English vs Other), but different F ratios for within subject effects (e.g., Class(animate vs inanimate), aov F-ratios being consistently smaller and more conservative. The interaction of the within-between factors again yields identical terms. I am stumped. HOw can this be? Any suggestion? Here are more details:

I was analyzing the RTs lexdec data base from the languageR package. I did some minor data adjustments. E.g., To get a sense for RTs I reversed the log transform exp(lexdec$RT) then removed Error responses and RT outliers. Using ddply I obtained condition means for NativeLanguage and Class for each subject (the data frame is shown at the bottom ob my post). Analyzing these data, I obtained different aov and STATISTICA summaries. Specifically for the within-subjects factor Class was p~.18 with aov and p~.06 with STATISTICA and SPSS, with larger Class SS values shown by the two commercial packages (569) than by aov (301). I've tried to make the two anova outputs (shown below) look transparent but it appears that the posting format does not match the format shown in the question window. > C1.anova <- aov(RT ~ (Class * NativeLanguage) + + Error(Subject/Class) + (NativeLanguage), data=C1 ) > summary(C1.anova) Error: Subject Df Sum Sq Mean Sq F value Pr(>F) NativeLanguage 1 81413 81413 6.2973 0.02131 * Residuals 19 245637 12928 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Error: Subject:Class Df Sum Sq Mean Sq F value Pr(>F) Class 1 301.51 301.51 1.9661 0.176994 Class:NativeLanguage 1 2175.86 2175.86 14.1880 0.001305 ** Residuals 19 2913.83 153.36 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1  STATISTICA  SS DF MS F p Intercept 15387240 1 15387240 1190.202 0.000000 NativeLanguage 81413 1 81413 6.297 0.021311 Error 245637 19 12928 Class 569 1 569 3.709 0.069217 Class*NativeLanguage 2176 1 2176 14.188 0.001305 Error 2914 19 153  Data:  Subject Class NativeLanguage RT 1 A1 animal English 557.6410 2 A1 plant English 548.4687 3 A2 animal English 533.4737 4 A2 plant English 511.7941 5 A3 animal Other 598.9545 6 A3 plant Other 602.4118 7 C animal English 562.8864 8 C plant English 560.0588 9 D animal Other 630.1464 10 D plant Other 604.0286 11 I animal Other 542.1219 12 I plant Other 533.1666 13 J animal Other 565.4324 14 J plant Other 513.2333 15 K animal English 492.4500 16 K plant English 517.7333 17 M1 animal English 481.8372 18 M1 plant English 497.9687 19 M2 animal Other 671.6666 20 M2 plant Other 655.8750 21 P animal Other 640.7209 22 P plant Other 610.0286 23 R1 animal English 552.9744 24 R1 plant English 545.4242 25 R2 animal English 636.8864 26 R2 plant English 675.1714 27 R3 animal English 607.8572 28 R3 plant English 614.9428 29 S animal English 599.9285 30 S plant English 586.6286 31 T1 animal English 580.0500 32 T1 plant English 583.2857 33 T2 animal Other 892.5526 34 T2 plant Other 862.1000 35 V animal Other 736.2619 36 V plant Other 718.3529 37 W1 animal English 517.0465 38 W1 plant English 539.2727 39 W2 animal English 639.1363 40 W2 plant English 666.7143 41 Z animal Other 725.3750 42 Z plant Other 706.2069  • FYI, the formula supplied to aov() has redundant terms; if you have Class*NativeLanguage, you don't need +NativeLanguage Oct 6, 2011 at 16:12 • This question also addresses the issue of Type III SS in R: stats.stackexchange.com/q/4544/3601 Oct 7, 2011 at 13:56 2 Answers This may be because your between-groups variable, NativeLanguage, is unbalanced (12 English, 9 Other), in which case the type of Sums-of-Squares employed is going to affect the F values. By default, aov() uses Type 1 sums of squares, which isn't recommended with unbalanced designs. Instead, use the ezANOVA() function from the ez package: my_anova = ezANOVA( data = C1 , dv = .(RT) , wid = .(Subject) , within = .(Class) , between = .(NativeLanguage) , type = 3 #SPSS uses type 3 Sums-of-Squares , observed = .(NativeLanguage) #ensures appropriate effect size is computed ) #note warning about data imbalance print(my_anova)  This yields the results table: $ANOVA
Effect DFn DFd         F           p p<.05          ges
2       NativeLanguage   1  19  6.297322 0.021311506     * 0.2451177279
3                Class   1  19  1.976662 0.175885891       0.0009118545
4 NativeLanguage:Class   1  19 14.187926 0.001305329     * 0.0065510116


Which strangely still has a different report for the Class effect than what you're getting from SPSS/Statistica. Adding a detailed=TRUE argument to the ezANOVA() call above gives us a slightly more detailed results table (including the intercept and sums of squares):

\$ANOVA
Effect DFn DFd          SSn        SSd          F            p p<.05          ges
1          (Intercept)   1  19 7717585.5514 245636.967 596.954632 8.136647e-16     * 0.9587389575
2       NativeLanguage   1  19   81413.4195 245636.967   6.297322 2.131151e-02     * 0.2451177279
3                Class   1  19     303.1399   2913.831   1.976662 1.758859e-01       0.0009118545
4 NativeLanguage:Class   1  19    2175.8535   2913.831  14.187926 1.305329e-03     * 0.0065510116


This shows that the mismatch lies in the SSn for the class effect; ezANOVA (which uses car::Anova()) obtains an SSn of 303ish whereas SPSS/Statistica obtain an SSn of 569.

• Ok, I had to leave for lunch and thought the type 3 sum of squares was THE answer. However, I just had a chance to do the ezANOVA, as suggested. It ran, as specified, but I still get the same F and p effects as I do get with the aov function and the within subjects effect differs from the F and p that SPSS gives me. Oct 6, 2011 at 19:05
• Yes, I updated my answer subsequent to its first posting to note that there's still a disparity that I can't track down without access to SPSS. Oct 6, 2011 at 20:51

As Mike already pointed out, the difference is due to the choice for the sum of squares (III for SPSS, I for R). The important thing to add is that type III SS require a specific coding scheme for categorical variables: it needs to be of the sum-to-zero kind, e.g., effect coding or Helmert coding. R's default is dummy coding, where the coefficients do not sum to zero (compare contr.sum(4), contr.helmert(4) with contr.treatment(4)). This will cause wrong results for type III SS.

I'll use car's Anova() function for type III SS. It requires data in wide format.

> Cw <- reshape(C1, direction="wide", v.names="RT", timevar="Class",
+               idvar=c("Subject", "NativeLanguage"))

> options(contrasts=c("contr.sum", "contr.poly"))              # switch to effect coding
> library(car)                                                 # for Anova()
> fitSPFpq <- lm(cbind(RT.animal, RT.plant) ~ NativeLanguage, data=Cw)
> inSPFpq  <- data.frame(Class=gl(2, 1))
> summary(Anova(fitSPFpq, idata=inSPFpq, idesign=~Class, type="III"), multivariate=FALSE)
Univariate Type III Repeated-Measures ANOVA Assuming Sphericity

SS num Df Error SS den Df         F    Pr(>F)
(Intercept)          15387240      1   245637     19 1190.2018 < 2.2e-16 ***
NativeLanguage          81413      1   245637     19    6.2973  0.021312 *
Class                     569      1     2914     19    3.7090  0.069216 .
NativeLanguage:Class     2176      1     2914     19   14.1879  0.001305 **


The results match those from SPSS and Statistica also for the Class effect.

(As a sidenote: some regard the dependence of type III SS in unbalanced Anova designs on the - arbitrary - choice for a coding scheme as a significant downside. The dependence comes from the chosen model comparisons which do not follow the principle of marginality.)

• @alwin: caracal's answer explains the difference well, but it should be noted that this is more than a technical problem of getting the softare to agree; you should think about your scientific questions and decide which is more appropriate. In most cases that I've done, the Type I or II (which are the same here, I think) is more appropriate than Type III, but you should think about your questions and decide for yourself. Oct 7, 2011 at 13:52
• @Aaron Good points about the equivalence of type I and type II tests in this case, and about the different SS types effectively testing different hypotheses! Oct 7, 2011 at 16:24
• Thank you very much, Caracal and Aaron. This additional information is extremely helpful. My sense is that Type 1 is more appropriate in this case. In fact, when I computed NativeLanguage and Class condition means for each word of the data set and then used 'Words' as random factor (often done in psycholinguistic studies), then the F value for Class was less than 1 -- meaning Class has no real effect. Oct 7, 2011 at 17:54