# ANOVA rejects null hypothesis while Tukey doesn't reject

I found an example where according to ANOVA, the means of the groups aren't all the same but Multiple comparison Tukey-Kramer test shows that they are the same pairwise.

Is there an example when the opposite is true: according to ANOVA, the means of the groups are all the same but Multiple comparison Tukey-Kramer test shows that at least in one pair there are different means.

If you could point out to research on this subject, I would highly appreciate it.

• Which "Tukey test" are you referring to? The HSD?
– whuber
Feb 26, 2021 at 19:08
• Multiple comparison Tukey-Kramer test that uses Studentized critical range
– Vika
Feb 26, 2021 at 20:09
• Two different criteria are used for the ANOVA and for Tukey HSD. Occasionally, it can happen that the former rejects (perhaps just barely at 5% level), while the latter finds no differences among levels--or, more commonly, detects that levels with largest and smallest mean differ, but doesn't resolve intermediate differences. 'Equal to' is a transitive relation; 'not significantly different from' is not. Feb 26, 2021 at 23:18

## 1 Answer

Comment continued: Here is a slight modification of the link in my comment, in which ANOVA rejects at 3% and Tukey HSD rejects nothing at 5%.

Simulated data:

set.seed(226)
sg=3.4; n = 15
x1 = rnorm(n, 101, sg)
x2 = rnorm(n, 101, sg)
x3 = rnorm(n, 102, sg)
x4 = rnorm(n, 103, sg)
x5 = rnorm(n, 103, sg)
round(rowMeans(rbind(x1,x2,x3,x4,x5)),3)
x1      x2      x3      x4      x5
100.262 100.208 102.467 101.546 103.592


ANOVA:

x = c(x1,x2,x3,x4,x5)
g = as.factor(rep(1:5,each=n))
aov.out = aov(lm(x ~ g))
summary(aov.out)
Df Sum Sq Mean Sq F value Pr(>F)
g            4  126.7   31.68   2.867 0.0293 *
Residuals   70  773.5   11.05
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


Tukey HSD:

TukeyHSD(aov.out)
Tukey multiple comparisons of means
95% family-wise confidence level

Fit: aov(formula = lm(x ~ g))

\$g
diff         lwr      upr     p adj
2-1 -0.05425495 -3.45301738 3.344507 0.9999991
3-1  2.20535061 -1.19341183 5.604113 0.3722334
4-1  1.28353622 -2.11522621 4.682299 0.8274004
5-1  3.32946337 -0.06929906 6.728226 0.0576539
3-2  2.25960556 -1.13915688 5.658368 0.3474994
4-2  1.33779117 -2.06097126 4.736554 0.8048222
5-2  3.38371832 -0.01504411 6.782481 0.0515827
4-3 -0.92181438 -4.32057682 2.476948 0.9412335
5-3  1.12411276 -2.27464967 4.522875 0.8858393
5-4  2.04592715 -1.35283529 5.444690 0.4493883


In my experience, this kind of 'discrepancy' between ANOVA and Tukey HSD most often occurs in underpowered studies.

Power:

The following power and sample size procedure from Minitab suggests that an ANOVA might might not detect differences of $$2$$ among five levels with $$n=15$$ replications when $$\sigma = 3.4.$$

Power and Sample Size

One-way ANOVA

α = 0.05  Assumed standard deviation = 3.4

Factors: 1  Number of levels: 5

Maximum  Sample
Difference    Size     Power
2      15  0.201956

The sample size is for each level.


Note: You asked about the opposite discrepancy where Tukey finds differences when ANOVA does not. Strictly speaking, one should not even do Tukey ad hoc tests unless the ANOVA rejects. So this should never be observed---under good practice.

I did some simulations (with different seeds) to investigate how often this would be seen by someone 'breaking the rules'. My impression is that it is much less likely for Tukey HSD to 'discover' what ANOVA fails to 'promise', than the other way around. But it does happen. Here is partial output for one such instance (with $$\sigma = 3.0).$$

round(rowMeans(rbind(x1,x2,x3,x4,x5)),3)
x1      x2      x3      x4      x5
101.137  99.991 102.149 101.601 102.854

...

summary(aov.out)
Df Sum Sq Mean Sq F value Pr(>F)
g            4   69.9  17.481   2.288 0.0684 .
Residuals   70  534.7   7.639

...

4-2  1.6096132 -1.21642062 4.435647 0.5056652
5-2  2.8626855  0.03665162 5.688719 0.0456020  <---
4-3 -0.5479727 -3.37400653 2.278061 0.9824543


• Different test have different power and that can lead to different results. Feb 27, 2021 at 1:26