18
$\begingroup$

I performed with R an ANOVA and I got significant differences. However when checking which pairs were significantly different using the Tukey's procedure I did not get any of them. How can this be possible?

Here is the code:

fit5_snow<- lm(Response ~ Stimulus, data=audio_snow)
anova(fit5_snow)

> anova(fit5_snow)
Analysis of Variance Table

Response: Response
          Df Sum Sq Mean Sq F value  Pr(>F)  
Stimulus   5  73.79 14.7578  2.6308 0.02929 *
Residuals 84 471.20  5.6095                  
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1 

df<-df.residual(fit5_snow)
MSerror<-deviance(fit5_snow)/df

comparison <-  HSD.test(audio_snow$Response, audio_snow$Stimulus, df, MSerror, group=FALSE)

> comparison <-  HSD.test(audio_snow$Response, audio_snow$Stimulus, df, MSerror, group=FALSE)

Study:

HSD Test for audio_snow$Response 

Mean Square Error:  5.609524 

audio_snow$Stimulus,  means

                audio_snow.Response   std.err replication
snow_dry_leaves            4.933333 0.6208034          15
snow_gravel                6.866667 0.5679258          15
snow_metal                 6.333333 0.5662463          15
snow_sand                  6.733333 0.5114561          15
snow_snow                  7.333333 0.5989409          15
snow_wood                  5.066667 0.7713110          15

alpha: 0.05 ; Df Error: 84 
Critical Value of Studentized Range: 4.124617 

Comparison between treatments means

                              Difference   pvalue sig        LCL      UCL
snow_gravel - snow_dry_leaves  1.9333333 0.232848     -0.5889913 4.455658
snow_metal - snow_dry_leaves   1.4000000 0.588616     -1.1223246 3.922325
snow_sand - snow_dry_leaves    1.8000000 0.307012     -0.7223246 4.322325
snow_snow - snow_dry_leaves    2.4000000 0.071587   . -0.1223246 4.922325
snow_wood - snow_dry_leaves    0.1333333 0.999987     -2.3889913 2.655658
snow_gravel - snow_metal       0.5333333 0.989528     -1.9889913 3.055658
snow_gravel - snow_sand        0.1333333 0.999987     -2.3889913 2.655658
snow_snow - snow_gravel        0.4666667 0.994348     -2.0556579 2.988991
snow_gravel - snow_wood        1.8000000 0.307012     -0.7223246 4.322325
snow_sand - snow_metal         0.4000000 0.997266     -2.1223246 2.922325
snow_snow - snow_metal         1.0000000 0.855987     -1.5223246 3.522325
snow_metal - snow_wood         1.2666667 0.687424     -1.2556579 3.788991
snow_snow - snow_sand          0.6000000 0.982179     -1.9223246 3.122325
snow_sand - snow_wood          1.6666667 0.393171     -0.8556579 4.188991
snow_snow - snow_wood          2.2666667 0.103505     -0.2556579 4.788991
$\endgroup$
2
$\begingroup$

Why should it not be possible?

The overall test and the pairwise tests ask different questions, so they can get different answers.

$\endgroup$
  • 1
    $\begingroup$ Could you please say more. $\endgroup$ – rolando2 Oct 8 '11 at 1:54
  • 2
    $\begingroup$ The overall ANOVA asks a question about the whole independent variable and its relation (or lack thereof) with the dependent variable. The pairwise comparisons ask about differences among pairs. Then the p-value looks at the statistical sig. of each of these, with the pairwise adjusted for multiple comparisons (in this case, using Tukey's HSD methods). $\endgroup$ – Peter Flom - Reinstate Monica Oct 8 '11 at 10:31
  • 1
    $\begingroup$ thanks, Peter. Maybe it's less that they ask "different questions" and more the adjustment for multiple comparisons that accounts for the different result. $\endgroup$ – rolando2 Oct 8 '11 at 11:37
17
$\begingroup$

This is mainly due to the sensitivity of ANOVA (greater than the pairwise test sensitivity). Then, ANOVA detect lower variability around mean when pairwise test hardly distinguishes between the pair's mean. The analysis must focus on the differences, and you can be more flexible on the post-hoc analysis, having in mind that you just encountered that there exists differences on the mean. Remember to check the ANOVA assumptions.

On the other hand, there are some topics concerning the use of pairwise test without using ANOVA: Do we need a global test before post hoc tests?

$\endgroup$
  • $\begingroup$ Definitely do not need a global test before doing Tukey hsd comparisons since hsd controls the Type I error rate. I hate to call them post hocs, though, because they should be planned a priori. $\endgroup$ – David Lane Feb 4 '17 at 22:31

protected by Community Feb 15 '17 at 20:11

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.