2
$\begingroup$

I found an example where according to ANOVA, the means of the groups aren't all the same but Multiple comparison Tukey-Kramer test shows that they are the same pairwise.

Is there an example when the opposite is true: according to ANOVA, the means of the groups are all the same but Multiple comparison Tukey-Kramer test shows that at least in one pair there are different means.

If you could point out to research on this subject, I would highly appreciate it.

$\endgroup$
3
  • $\begingroup$ Which "Tukey test" are you referring to? The HSD? $\endgroup$
    – whuber
    Feb 26, 2021 at 19:08
  • 1
    $\begingroup$ Multiple comparison Tukey-Kramer test that uses Studentized critical range $\endgroup$
    – Vika
    Feb 26, 2021 at 20:09
  • $\begingroup$ Two different criteria are used for the ANOVA and for Tukey HSD. Occasionally, it can happen that the former rejects (perhaps just barely at 5% level), while the latter finds no differences among levels--or, more commonly, detects that levels with largest and smallest mean differ, but doesn't resolve intermediate differences. 'Equal to' is a transitive relation; 'not significantly different from' is not. $\endgroup$
    – BruceET
    Feb 26, 2021 at 23:18

1 Answer 1

1
$\begingroup$

Comment continued: Here is a slight modification of the link in my comment, in which ANOVA rejects at 3% and Tukey HSD rejects nothing at 5%.

Simulated data:

set.seed(226)
sg=3.4; n = 15
x1 = rnorm(n, 101, sg)
x2 = rnorm(n, 101, sg)
x3 = rnorm(n, 102, sg)
x4 = rnorm(n, 103, sg)
x5 = rnorm(n, 103, sg)
round(rowMeans(rbind(x1,x2,x3,x4,x5)),3)
     x1      x2      x3      x4      x5 
100.262 100.208 102.467 101.546 103.592 

ANOVA:

x = c(x1,x2,x3,x4,x5)
g = as.factor(rep(1:5,each=n))
aov.out = aov(lm(x ~ g))
summary(aov.out)
            Df Sum Sq Mean Sq F value Pr(>F)  
g            4  126.7   31.68   2.867 0.0293 *
Residuals   70  773.5   11.05                 
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Tukey HSD:

TukeyHSD(aov.out)
  Tukey multiple comparisons of means
    95% family-wise confidence level

Fit: aov(formula = lm(x ~ g))

$g
           diff         lwr      upr     p adj
2-1 -0.05425495 -3.45301738 3.344507 0.9999991
3-1  2.20535061 -1.19341183 5.604113 0.3722334
4-1  1.28353622 -2.11522621 4.682299 0.8274004
5-1  3.32946337 -0.06929906 6.728226 0.0576539
3-2  2.25960556 -1.13915688 5.658368 0.3474994
4-2  1.33779117 -2.06097126 4.736554 0.8048222
5-2  3.38371832 -0.01504411 6.782481 0.0515827
4-3 -0.92181438 -4.32057682 2.476948 0.9412335
5-3  1.12411276 -2.27464967 4.522875 0.8858393
5-4  2.04592715 -1.35283529 5.444690 0.4493883

In my experience, this kind of 'discrepancy' between ANOVA and Tukey HSD most often occurs in underpowered studies.

Power:

The following power and sample size procedure from Minitab suggests that an ANOVA might might not detect differences of $2$ among five levels with $n=15$ replications when $\sigma = 3.4.$

Power and Sample Size 

One-way ANOVA

α = 0.05  Assumed standard deviation = 3.4

Factors: 1  Number of levels: 5

   Maximum  Sample
Difference    Size     Power
         2      15  0.201956

The sample size is for each level.

enter image description here

Note: You asked about the opposite discrepancy where Tukey finds differences when ANOVA does not. Strictly speaking, one should not even do Tukey ad hoc tests unless the ANOVA rejects. So this should never be observed---under good practice.

I did some simulations (with different seeds) to investigate how often this would be seen by someone 'breaking the rules'. My impression is that it is much less likely for Tukey HSD to 'discover' what ANOVA fails to 'promise', than the other way around. But it does happen. Here is partial output for one such instance (with $\sigma = 3.0).$

round(rowMeans(rbind(x1,x2,x3,x4,x5)),3)
     x1      x2      x3      x4      x5 
101.137  99.991 102.149 101.601 102.854 

...

summary(aov.out)
            Df Sum Sq Mean Sq F value Pr(>F)  
g            4   69.9  17.481   2.288 0.0684 .
Residuals   70  534.7   7.639 

...            

4-2  1.6096132 -1.21642062 4.435647 0.5056652
5-2  2.8626855  0.03665162 5.688719 0.0456020  <---
4-3 -0.5479727 -3.37400653 2.278061 0.9824543
  
$\endgroup$
1
  • $\begingroup$ Different test have different power and that can lead to different results. $\endgroup$
    – user54285
    Feb 27, 2021 at 1:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.