# Why does post hoc test on pair of group means become significant when looking at subset of levels of factor in ANOVA?

I have something that seems to me an incongruence in ANOVA post hoc tests, and I would like to have an explanation. Basically I performed an ANOVA at the global level on my data and then I performed a post hoc test using both LSD and Tukey's HSD procedure. First I did the ANOVA on the whole data set and post hoc tests showed some pairs significantly different. Afterwards I subdivided the data set considering only some stimuli and I performed the post hoc test on them. In this second case some of the pairs that were non-significant in the "global" post hoc test become significant in the subset case. How does this happen?

More in detail I show you what I did in R.

fit5<- lm(Response ~ Stimulus, data=scrd)
library(agricolae)
df<-df.residual(fit5)
MSerror<-deviance(fit5)/df

comparison <- LSD.test(scrd$Response, scrd$Stimulus, df,

Study:

LSD t Test for scrd$Response P value adjustment method: bonferroni Mean Square Error: 4.292088 scrd$Stimulus,  means and individual ( 95 %) CI

scrd.Response   std.err replication      LCL      UCL
dry_leaves_dry_leaves      6.833333 0.7768754          12 5.306018 8.360649
dry_leaves_gravel          6.750000 0.5383054          12 5.691706 7.808294
dry_leaves_metal           3.250000 0.5093817          12 2.248570 4.251430
dry_leaves_sand            6.583333 0.5701984          12 5.462339 7.704328
...
...

alpha: 0.05 ; Df Error: 396
Critical Value of t: 3.987986

Comparison between treatments means

Difference   pvalue sig         LCL      UCL
dry_leaves_dry_leaves - dry_leaves_gravel 0.08333333 1.000000     -3.28963527 3.456302
dry_leaves_dry_leaves - dry_leaves_metal  3.58333333 0.017792   *  0.21036473 6.956302
dry_leaves_dry_leaves - dry_leaves_sand   0.25000000 1.000000     -3.12296860 3.622969
dry_leaves_dry_leaves - dry_leaves_snow   1.41666667 1.000000     -1.95630194 4.789635
dry_leaves_dry_leaves - dry_leaves_wood   1.83333333 1.000000     -1.53963527 5.206302
dry_leaves_dry_leaves - gravel_dry_leaves 0.58333333 1.000000     -2.78963527 3.956302
dry_leaves_dry_leaves - gravel_gravel     0.41666667 1.000000     -2.95630194 3.789635
dry_leaves_dry_leaves - gravel_metal      3.66666667 0.011649   *  0.29369806 7.039635
...
...
wood_sand - wood_snow                     1.08333333 1.000000     -2.28963527 4.456302
wood_wood - wood_sand                     3.00000000 0.274864     -0.37296860 6.372969
wood_wood - wood_snow                     4.08333333 0.001244  **  0.71036473 7.456302


Now I extract from the whole dataset a subdataset and I perform the ANOVA and post-hoc test on it:

# Row extraction
audio_wood <- subset(scrd, Audio == "wood")

#-------wood-------#

fit5_wood<- lm(Response ~ Stimulus, data=audio_wood)
anova(fit5_wood)

> anova(fit5_wood)
Analysis of Variance Table

Response: Response
Df Sum Sq Mean Sq F value    Pr(>F)
Stimulus   5 236.24  47.247  12.604 1.333e-08 ***
Residuals 66 247.42   3.749
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1


Here I do the post hoc test

df<-df.residual(fit5_wood)
MSerror<-deviance(fit5_wood)/df
comparison <- LSD.test(audio_wood$Response, audio_wood$Stimulus, df,

Study:

LSD t Test for audio_wood$Response P value adjustment method: bonferroni Mean Square Error: 3.748737 audio_wood$Stimulus,  means and individual ( 95 %) CI

audio_wood.Response   std.err replication      LCL      UCL
wood_dry_leaves            3.916667 0.5143153          12 2.889803 4.943530
wood_gravel                2.916667 0.3361622          12 2.245497 3.587836
wood_metal                 7.333333 0.4143877          12 6.505982 8.160685
wood_sand                  4.000000 0.7487363          12 2.505100 5.494900
wood_snow                  2.916667 0.7329717          12 1.453242 4.380092
wood_wood                  7.000000 0.4767313          12 6.048175 7.951825

alpha: 0.05 ; Df Error: 66
Critical Value of t: 3.045792

Comparison between treatments means

Difference   pvalue sig        LCL      UCL
wood_dry_leaves - wood_gravel 1.00000000 1.000000     -1.4075050 3.407505
wood_metal - wood_dry_leaves  3.41666667 0.000798 ***  1.0091617 5.824172
wood_sand - wood_dry_leaves   0.08333333 1.000000     -2.3241716 2.490838
wood_dry_leaves - wood_snow   1.00000000 1.000000     -1.4075050 3.407505
wood_wood - wood_dry_leaves   3.08333333 0.003409  **  0.6758284 5.490838
wood_metal - wood_gravel      4.41666667 0.000007 ***  2.0091617 6.824172
wood_sand - wood_gravel       1.08333333 1.000000     -1.3241716 3.490838
wood_gravel - wood_snow       0.00000000 1.000000     -2.4075050 2.407505
wood_wood - wood_gravel       4.08333333 0.000036 ***  1.6758284 6.490838
wood_metal - wood_sand        3.33333333 0.001155  **  0.9258284 5.740838
wood_metal - wood_snow        4.41666667 0.000007 ***  2.0091617 6.824172
wood_metal - wood_wood        0.33333333 1.000000     -2.0741716 2.740838
wood_sand - wood_snow         1.08333333 1.000000     -1.3241716 3.490838
wood_wood - wood_sand         3.00000000 0.004843  **  0.5924950 5.407505
wood_wood - wood_snow         4.08333333 0.000036 ***  1.6758284 6.490838


As you can notice, the pair wood_wood - wood_sand which was not significant in the previous global pst hoc test, now is significant.

Which of the two analysis I have to believe? And why?

-