example where comparison of three mean anova and t-test have different results

Question

suppose to have sample from 3 groups A,B,C. The hypothesis H0: the mean of the 3 groups is the same can be tested using 3 independent t test.

test1: mean(A)=mean(B) level 0.05
test2: mean(B)=mean(C) level 0.05
test3: mean(A)=mean(C) level 0.05

It's known that we should prefer an ANOVA test because with the previous method there is an increasing risk of type 1 error.

I would like to have an example with simulated data where the first procedure lead us to an error while the ANOVA return the good result.

The best would be an R code to simulate the experiment.

Thanks!

Answer 1

The following code will replicate a situation where three groups are randomly generated from the same normal distributions, N(0,25). Here, one of the t-tests commits a Type I error that is not commited by ANOVA on the same data.

set.seed(270)
As = rnorm(5, mean = 0, sd = 5)
Bs = rnorm(5, mean = 0, sd = 5)
Cs = rnorm(5, mean = 0, sd = 5)

dat = data.frame(factor = c("A","A","A","A","A","B","B","B","B","B", "C","C","C","C","C"),
                 response = c(As, Bs, Cs))

summary(aov(response ~ factor, data = dat))
t.test(As, Bs)
t.test(Bs, Cs)
t.test(As, Cs)

Anova output:

            Df Sum Sq Mean Sq F value Pr(>F)
factor       2  88.88   44.44   2.233   0.15
Residuals   12 238.82   19.90 

T-test output:

data:  As and Bs
t = -0.9327, df = 7.42, p-value = 0.3803

data:  Bs and Cs
t = -1.0132, df = 4.968, p-value = 0.3577

data:  As and Cs
t = -2.7043, df = 5.666, p-value = 0.03746*

So the t-test detects a significant difference between groups A and C ($\alpha = 0.05$) committing a Type I error. ANOVA, correctly, suggests there is not enough evidence of a significance difference between the groups.