How can I get a significant overall ANOVA but no significant pairwise differences with Tukey's procedure?

Question

I performed with R an ANOVA and I got significant differences. However when checking which pairs were significantly different using the Tukey's procedure I did not get any of them. How can this be possible?

Here is the code:

fit5_snow<- lm(Response ~ Stimulus, data=audio_snow)
anova(fit5_snow)

> anova(fit5_snow)
Analysis of Variance Table

Response: Response
          Df Sum Sq Mean Sq F value  Pr(>F)  
Stimulus   5  73.79 14.7578  2.6308 0.02929 *
Residuals 84 471.20  5.6095                  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 

df<-df.residual(fit5_snow)
MSerror<-deviance(fit5_snow)/df

comparison <-  HSD.test(audio_snow$Response, audio_snow$Stimulus, df, MSerror, group=FALSE)

> comparison <-  HSD.test(audio_snow$Response, audio_snow$Stimulus, df, MSerror, group=FALSE)

Study:

HSD Test for audio_snow$Response 

Mean Square Error:  5.609524 

audio_snow$Stimulus,  means

                audio_snow.Response   std.err replication
snow_dry_leaves            4.933333 0.6208034          15
snow_gravel                6.866667 0.5679258          15
snow_metal                 6.333333 0.5662463          15
snow_sand                  6.733333 0.5114561          15
snow_snow                  7.333333 0.5989409          15
snow_wood                  5.066667 0.7713110          15

alpha: 0.05 ; Df Error: 84 
Critical Value of Studentized Range: 4.124617 

Comparison between treatments means

                              Difference   pvalue sig        LCL      UCL
snow_gravel - snow_dry_leaves  1.9333333 0.232848     -0.5889913 4.455658
snow_metal - snow_dry_leaves   1.4000000 0.588616     -1.1223246 3.922325
snow_sand - snow_dry_leaves    1.8000000 0.307012     -0.7223246 4.322325
snow_snow - snow_dry_leaves    2.4000000 0.071587   . -0.1223246 4.922325
snow_wood - snow_dry_leaves    0.1333333 0.999987     -2.3889913 2.655658
snow_gravel - snow_metal       0.5333333 0.989528     -1.9889913 3.055658
snow_gravel - snow_sand        0.1333333 0.999987     -2.3889913 2.655658
snow_snow - snow_gravel        0.4666667 0.994348     -2.0556579 2.988991
snow_gravel - snow_wood        1.8000000 0.307012     -0.7223246 4.322325
snow_sand - snow_metal         0.4000000 0.997266     -2.1223246 2.922325
snow_snow - snow_metal         1.0000000 0.855987     -1.5223246 3.522325
snow_metal - snow_wood         1.2666667 0.687424     -1.2556579 3.788991
snow_snow - snow_sand          0.6000000 0.982179     -1.9223246 3.122325
snow_sand - snow_wood          1.6666667 0.393171     -0.8556579 4.188991
snow_snow - snow_wood          2.2666667 0.103505     -0.2556579 4.788991

Answer 1

Why should it not be possible?

The overall test and the pairwise tests ask different questions, so they can get different answers.

The overall ANOVA asks a question about the whole independent variable and its relation (or lack thereof) with the dependent variable. The pairwise comparisons ask about differences among pairs. Then the p-value looks at the statistical sig. of each of these, with the pairwise adjusted for multiple comparisons (in this case, using Tukey's HSD methods).

Answer 2

This is mainly due to the sensitivity of ANOVA (greater than the pairwise test sensitivity). Then, ANOVA detect lower variability around mean when pairwise test hardly distinguishes between the pair's mean. The analysis must focus on the differences, and you can be more flexible on the post-hoc analysis, having in mind that you just encountered that there exists differences on the mean. Remember to check the ANOVA assumptions.

On the other hand, there are some topics concerning the use of pairwise test without using ANOVA: Do we need a global test before post hoc tests?

Answer 3

Below is a copy from an answer to a duplicate question R Tukey HSD Anova: Anova significant, Tukey not?

Since this answer is not visible here, and there is no clear link, I create this copy.

---

The relationship between the p-values for the F-test and Tukey HSD test is not one-to-one. (even though both test, indirectly, equality of means $\mu_1=\mu_2=\mu_3$)

This is because, for a given distance between the smallest and the largest mean (defining the smallest p-value in Tukey's HSD test), the between group variance (defining the p-value in ANOVA) still depends on the position of the middle mean. The between groups variance is largest when two groups of the three groups cluster together at one end, rather than when the three groups are equally distributed.

For instance: the means 0, 0.5 and 1 have a smaller between groups variance than the means 0, 1 and 1. But the largest distance (between the outside groups) is the same. That means the smallest p-value in Tukey HSD test will not be different for those two cases while the ANOVA p-value does differ.

So for the experiments with the 5% largest significant differences, you do not get the 5% largest F-scores (or vice versa). It depends on the distribution of the groups and two small p-values in the Tukey test just above 5% can make a F-test with a p-value below 5%. (this becomes even stronger when you have a larger number of groups)

---

The below image is made from a simulation of 1000 draws for three groups of size 50 from a standard normal distribution.

It compares

• the criteria for the Tukey HSD test (showing the smallest and second smallest p-values on the x- and y-axis with two vertical lines at 0.05 and 0.1)
• with the criteria for the F-test (the red dots have p-value below 0.05, the green dots have p-value above 0.05 and below 0.1, black dots are above 0.1).

The p-value for the F-test does not align with the smallest p-value for the Tukey HSD test. The p-value for the F-test can be both higher or lower than the p-value for the lowest p-value in the Tukey HSD test, depending on the other p-values in the Tukey HSD test (this is analogous with the earlier mentioned difference between clustered distribution of group means and even distribution of group means).

Note that both the Tukey HSD test (the lowest p-value) and the F-test reject their associated hypothesis for a different fraction of the 1000 simulated experiments, but the sizes of the fractions are equivalent and both correspond to the same type-I error rate.