3
$\begingroup$

I have something that seems to me an incongruence in ANOVA post hoc tests, and I would like to have an explanation. Basically I performed an ANOVA at the global level on my data and then I performed a post hoc test using both LSD and Tukey's HSD procedure. First I did the ANOVA on the whole data set and post hoc tests showed some pairs significantly different. Afterwards I subdivided the data set considering only some stimuli and I performed the post hoc test on them. In this second case some of the pairs that were non-significant in the "global" post hoc test become significant in the subset case. How does this happen?

More in detail I show you what I did in R.

fit5<- lm(Response ~ Stimulus, data=scrd)
library(agricolae)
df<-df.residual(fit5)
MSerror<-deviance(fit5)/df

comparison <- LSD.test(scrd$Response, scrd$Stimulus, df, 
                 MSerror, group=FALSE, p.adj="bonferroni")

Study:

LSD t Test for scrd$Response 
P value adjustment method: bonferroni 

Mean Square Error:  4.292088 

scrd$Stimulus,  means and individual ( 95 %) CI

                      scrd.Response   std.err replication      LCL      UCL
dry_leaves_dry_leaves      6.833333 0.7768754          12 5.306018 8.360649
dry_leaves_gravel          6.750000 0.5383054          12 5.691706 7.808294
dry_leaves_metal           3.250000 0.5093817          12 2.248570 4.251430
dry_leaves_sand            6.583333 0.5701984          12 5.462339 7.704328
...
...

alpha: 0.05 ; Df Error: 396
Critical Value of t: 3.987986 

Comparison between treatments means

                                          Difference   pvalue sig         LCL      UCL
dry_leaves_dry_leaves - dry_leaves_gravel 0.08333333 1.000000     -3.28963527 3.456302
dry_leaves_dry_leaves - dry_leaves_metal  3.58333333 0.017792   *  0.21036473 6.956302
dry_leaves_dry_leaves - dry_leaves_sand   0.25000000 1.000000     -3.12296860 3.622969
dry_leaves_dry_leaves - dry_leaves_snow   1.41666667 1.000000     -1.95630194 4.789635
dry_leaves_dry_leaves - dry_leaves_wood   1.83333333 1.000000     -1.53963527 5.206302
dry_leaves_dry_leaves - gravel_dry_leaves 0.58333333 1.000000     -2.78963527 3.956302
dry_leaves_dry_leaves - gravel_gravel     0.41666667 1.000000     -2.95630194 3.789635
dry_leaves_dry_leaves - gravel_metal      3.66666667 0.011649   *  0.29369806 7.039635
...
...
wood_sand - wood_snow                     1.08333333 1.000000     -2.28963527 4.456302
wood_wood - wood_sand                     3.00000000 0.274864     -0.37296860 6.372969
wood_wood - wood_snow                     4.08333333 0.001244  **  0.71036473 7.456302

Now I extract from the whole dataset a subdataset and I perform the ANOVA and post-hoc test on it:

# Row extraction
audio_wood <- subset(scrd, Audio == "wood")

#-------wood-------#

fit5_wood<- lm(Response ~ Stimulus, data=audio_wood)
anova(fit5_wood)


> anova(fit5_wood)
Analysis of Variance Table

Response: Response
          Df Sum Sq Mean Sq F value    Pr(>F)    
Stimulus   5 236.24  47.247  12.604 1.333e-08 ***
Residuals 66 247.42   3.749                      
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Here I do the post hoc test

df<-df.residual(fit5_wood)
MSerror<-deviance(fit5_wood)/df
comparison <- LSD.test(audio_wood$Response, audio_wood$Stimulus, df,
                 MSerror, group=FALSE, p.adj="bonferroni")

Study:

LSD t Test for audio_wood$Response 
P value adjustment method: bonferroni 

Mean Square Error:  3.748737 

audio_wood$Stimulus,  means and individual ( 95 %) CI

                audio_wood.Response   std.err replication      LCL      UCL
wood_dry_leaves            3.916667 0.5143153          12 2.889803 4.943530
wood_gravel                2.916667 0.3361622          12 2.245497 3.587836
wood_metal                 7.333333 0.4143877          12 6.505982 8.160685
wood_sand                  4.000000 0.7487363          12 2.505100 5.494900
wood_snow                  2.916667 0.7329717          12 1.453242 4.380092
wood_wood                  7.000000 0.4767313          12 6.048175 7.951825

alpha: 0.05 ; Df Error: 66
Critical Value of t: 3.045792 

Comparison between treatments means

                              Difference   pvalue sig        LCL      UCL
wood_dry_leaves - wood_gravel 1.00000000 1.000000     -1.4075050 3.407505
wood_metal - wood_dry_leaves  3.41666667 0.000798 ***  1.0091617 5.824172
wood_sand - wood_dry_leaves   0.08333333 1.000000     -2.3241716 2.490838
wood_dry_leaves - wood_snow   1.00000000 1.000000     -1.4075050 3.407505
wood_wood - wood_dry_leaves   3.08333333 0.003409  **  0.6758284 5.490838
wood_metal - wood_gravel      4.41666667 0.000007 ***  2.0091617 6.824172
wood_sand - wood_gravel       1.08333333 1.000000     -1.3241716 3.490838
wood_gravel - wood_snow       0.00000000 1.000000     -2.4075050 2.407505
wood_wood - wood_gravel       4.08333333 0.000036 ***  1.6758284 6.490838
wood_metal - wood_sand        3.33333333 0.001155  **  0.9258284 5.740838
wood_metal - wood_snow        4.41666667 0.000007 ***  2.0091617 6.824172
wood_metal - wood_wood        0.33333333 1.000000     -2.0741716 2.740838
wood_sand - wood_snow         1.08333333 1.000000     -1.3241716 3.490838
wood_wood - wood_sand         3.00000000 0.004843  **  0.5924950 5.407505
wood_wood - wood_snow         4.08333333 0.000036 ***  1.6758284 6.490838

As you can notice, the pair wood_wood - wood_sand which was not significant in the previous global pst hoc test, now is significant.

Which of the two analysis I have to believe? And why?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

A few thoughts:

  • Tukey's post-hoc test becomes more conservative on any one significance test as you increase the number of levels to your factor (thus looking at a subset of your levels should be more likely to show a significant effect)
  • The estimate of MS error can change between the overall and the subset, both in general, and particularly if homogeneity of variance is violated in the population from which the samples are drawn. Thus, if, for example, MS error gets larger in the overall analysis, then the overall analysis, would have less power to detect differences. But it could be the other way round.
  • In the subset analysis, your error degrees of freedom is less, which gives you less power to detect differences.

In general, if you are performing an ANOVA, and then performing follow-up tests, you would normally perform these on the entire set of levels of the factor in the ANOVA.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ ... and if your estimates of MSerror change considerably it probably means you violated the homogeneity of variances assumption anyway. $\endgroup$
    – russellpierce
    Commented Jul 9, 2013 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.