4
$\begingroup$

Suppose I create a dummy scenario as such:

> A <- rnorm(10000) 
> B <- rnorm(10000) 
> C <- rnorm(10000) 
> Y <- A*B + rnorm(10000,sd=0.1)

Doing a simple ANOVA correctly identifies that none of the variables are significantly predictive of the outcome:

> anova(lm(Y~A+B+C))
Analysis of Variance Table

Response: Y
            Df  Sum Sq Mean Sq F value Pr(>F)
A            1     1.5 1.54411  1.4209 0.2333
B            1     0.3 0.28909  0.2660 0.6060
C            1     1.6 1.62425  1.4946 0.2215
Residuals 9996 10862.8 1.08672    

But not let's say I decide to include the interaction terms:

> anova(lm(Y~A*B*C))
Analysis of Variance Table

Response: Y
           Df  Sum Sq Mean Sq    F value    Pr(>F)    
A            1     1.5     1.5 1.5281e+02 < 2.2e-16 ***
B            1     0.3     0.3 2.8610e+01  9.05e-08 ***
C            1     1.6     1.6 1.6074e+02 < 2.2e-16 ***
A:B          1 10761.8 10761.8 1.0650e+06 < 2.2e-16 ***
A:C          1     0.0     0.0 9.8700e-02    0.7534    
B:C          1     0.0     0.0 1.5062e+00    0.2197    
A:B:C        1     0.0     0.0 1.6790e-01    0.6820    
Residuals 9992   101.0     0.0                         
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

It has correctly identified the interaction between A and B as being the most significant, but now for some reason the individual terms A and B have also gained significance... and C which had nothing at all to do with creating the model is significant as well? Either I have not written the test correctly or I am completely misunderstanding how a Two-Way ANOVA with interaction terms works

Using a simple linear model gives expected results:

> summary(lm(Y~A*B*C))

Call:
lm(formula = Y ~ A * B * C)

Residuals:
     Min       1Q   Median       3Q      Max 
-0.29566 -0.06667 -0.00092  0.06665  0.33620 

Coefficients:
              Estimate Std. Error  t value Pr(>|t|)    
(Intercept) -0.0003212  0.0009707   -0.331    0.741    
A            0.0003483  0.0009613    0.362    0.717    
B            0.0003184  0.0009619    0.331    0.741    
C           -0.0003213  0.0009702   -0.331    0.741    
A:B          1.0008711  0.0009370 1068.214   <2e-16 ***
A:C         -0.0014855  0.0009588   -1.549    0.121    
B:C          0.0008860  0.0009561    0.927    0.354    
A:B:C       -0.0002489  0.0009085   -0.274    0.784    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.09705 on 9992 degrees of freedom
Multiple R-squared:  0.9913,    Adjusted R-squared:  0.9913 
F-statistic: 1.634e+05 on 7 and 9992 DF,  p-value: < 2.2e-16
$\endgroup$
3
$\begingroup$

This is because anova() and aov() in R use Type I sums of squares by default, instead of Type III sums of squares. The workaround is clumsy, but can be achieved thus:

library(car) #need to have package "car" installed
options(contrasts=c(unordered="contr.sum", ordered="contr.poly")) #set sum-to-zero contrast

Anova(lm(Y~A+B+C), type="III") #using the Anova() function from the car package
Anova(lm(Y~A*B*C), type="III") #main effects are not significant with Type III sums of squares

For example, I used your code to generate A, B, C, and Y, and used the code above to get my ANOVA tables, and I get the following:

> Anova(lm(Y~A+B+C), type="III")
Anova Table (Type III tests)

Response: Y
             Sum Sq   Df F value Pr(>F)
(Intercept)     1.3    1  1.2392 0.2657
A               0.9    1  0.9259 0.3360
B               2.0    1  2.0098 0.1563
C               0.0    1  0.0057 0.9397
Residuals   10111.8 9996  

Now with the interaction terms:

> Anova(lm(Y~A*B*C), type="III")
Anova Table (Type III tests)

Response: Y
            Sum Sq   Df    F value Pr(>F)    
(Intercept)    0.0    1 3.7080e-01 0.5426    
A              0.0    1 1.8000e-02 0.8932    
B              0.0    1 2.1904e+00 0.1389    
C              0.0    1 1.1700e-02 0.9139    
A:B         9980.5    1 1.0013e+06 <2e-16 ***
A:C            0.0    1 2.8640e-01 0.5925    
B:C            0.0    1 2.7710e-01 0.5986    
A:B:C          0.0    1 1.8800e-02 0.8909    
Residuals     99.6 9992                      
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

There is an entry about this in the R FAQ page, along with an alternative to get to the Type III sums of squares.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.