# How can I get a significant overall ANOVA but no significant pairwise differences with Tukey's procedure?

I performed with R an ANOVA and I got significant differences. However when checking which pairs were significantly different using the Tukey's procedure I did not get any of them. How can this be possible?

Here is the code:

fit5_snow<- lm(Response ~ Stimulus, data=audio_snow)
anova(fit5_snow)

> anova(fit5_snow)
Analysis of Variance Table

Response: Response
Df Sum Sq Mean Sq F value  Pr(>F)
Stimulus   5  73.79 14.7578  2.6308 0.02929 *
Residuals 84 471.20  5.6095
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

df<-df.residual(fit5_snow)
MSerror<-deviance(fit5_snow)/df

comparison <-  HSD.test(audio_snow$Response, audio_snow$Stimulus, df, MSerror, group=FALSE)

> comparison <-  HSD.test(audio_snow$Response, audio_snow$Stimulus, df, MSerror, group=FALSE)

Study:

HSD Test for audio_snow$Response Mean Square Error: 5.609524 audio_snow$Stimulus,  means

audio_snow.Response   std.err replication
snow_dry_leaves            4.933333 0.6208034          15
snow_gravel                6.866667 0.5679258          15
snow_metal                 6.333333 0.5662463          15
snow_sand                  6.733333 0.5114561          15
snow_snow                  7.333333 0.5989409          15
snow_wood                  5.066667 0.7713110          15

alpha: 0.05 ; Df Error: 84
Critical Value of Studentized Range: 4.124617

Comparison between treatments means

Difference   pvalue sig        LCL      UCL
snow_gravel - snow_dry_leaves  1.9333333 0.232848     -0.5889913 4.455658
snow_metal - snow_dry_leaves   1.4000000 0.588616     -1.1223246 3.922325
snow_sand - snow_dry_leaves    1.8000000 0.307012     -0.7223246 4.322325
snow_snow - snow_dry_leaves    2.4000000 0.071587   . -0.1223246 4.922325
snow_wood - snow_dry_leaves    0.1333333 0.999987     -2.3889913 2.655658
snow_gravel - snow_metal       0.5333333 0.989528     -1.9889913 3.055658
snow_gravel - snow_sand        0.1333333 0.999987     -2.3889913 2.655658
snow_snow - snow_gravel        0.4666667 0.994348     -2.0556579 2.988991
snow_gravel - snow_wood        1.8000000 0.307012     -0.7223246 4.322325
snow_sand - snow_metal         0.4000000 0.997266     -2.1223246 2.922325
snow_snow - snow_metal         1.0000000 0.855987     -1.5223246 3.522325
snow_metal - snow_wood         1.2666667 0.687424     -1.2556579 3.788991
snow_snow - snow_sand          0.6000000 0.982179     -1.9223246 3.122325
snow_sand - snow_wood          1.6666667 0.393171     -0.8556579 4.188991
snow_snow - snow_wood          2.2666667 0.103505     -0.2556579 4.788991

-
Can you give the data? –  ttnphns Oct 7 '11 at 15:07

Why should it not be possible?

The overall test and the pairwise tests ask different questions, so they can get different answers.

-
Could you please say more. –  rolando2 Oct 8 '11 at 1:54
The overall ANOVA asks a question about the whole independent variable and its relation (or lack thereof) with the dependent variable. The pairwise comparisons ask about differences among pairs. Then the p-value looks at the statistical sig. of each of these, with the pairwise adjusted for multiple comparisons (in this case, using Tukey's HSD methods). –  Peter Flom Oct 8 '11 at 10:31
thanks, Peter. Maybe it's less that they ask "different questions" and more the adjustment for multiple comparisons that accounts for the different result. –  rolando2 Oct 8 '11 at 11:37

This is mainly due to the sensitivity of ANOVA (greater than the pairwise test sensitivity). Then, ANOVA detect lower variability around mean when pairwise test hardly distinguishes between the pair's mean. The analysis must focus on the differences, and you can be more flexible on the post-hoc analysis, having in mind that you just encountered that there exists differences on the mean. Remember to check the ANOVA assumptions.

On the other hand, there are some topics concerning the use of pairwise test without using ANOVA: Do we need a global test before post hoc tests?

-
Ok, thanks a lot, I was not aware if this. –  Luca Oct 7 '11 at 16:01