R Tukey HSD Anova: Anova significant, Tukey not? [duplicate]

Question

i created a simple anova with three groups in R and get a significant Result. After that i do a Tukey HSD Test and now nothing is significant. How can that be?

Pr(>F): 0.0316

Group | p adj

1 - 2 | 0.0689014

2 - 3 | 0.0847163

3 - 1 | 0.8909709

Answer 1

The relationship between the p-values for the F-test and Tukey HSD test is not one-to-one. (even though both test, indirectly, equality of means $\mu_1=\mu_2=\mu_3$)

This is because, for a given distance between the smallest and the largest mean (defining the smallest p-value in Tukey's HSD test), the between group variance (defining the p-value in ANOVA) still depends on the position of the middle mean. The between groups variance is largest when two groups of the three groups cluster together at one end, rather than when the three groups are equally distributed.

For instance: the means 0, 0.5 and 1 have a smaller between groups variance than the means 0, 1 and 1. But the largest distance (between the outside groups) is the same. That means the smallest p-value in Tukey HSD test will not be different for those two cases while the ANOVA p-value does differ.

So for the experiments with the 5% largest significant differences, you do not get the 5% largest F-scores (or vice versa). It depends on the distribution of the groups and two small p-values in the Tukey test just above 5% can make a F-test with a p-value below 5%. (this becomes even stronger when you have a larger number of groups)

---

The below image is made from a simulation of 1000 draws for three groups of size 50 from a standard normal distribution.

It compares

• the criteria for the Tukey HSD test (showing the smallest and second smallest p-values on the x- and y-axis with two vertical lines at 0.05 and 0.1)
• with the criteria for the F-test (the red dots have p-value below 0.05, the green dots have p-value above 0.05 and below 0.1, black dots are above 0.1).

The p-value for the F-test does not align with the smallest p-value for the Tukey HSD test. The p-value for the F-test can be both higher or lower than the p-value for the lowest p-value in the Tukey HSD test, depending on the other p-values in the Tukey HSD test (this is analogous with the earlier mentioned difference between clustered distribution of group means and even distribution of group means).

Note that both the Tukey HSD test (the lowest p-value) and the F-test reject their associated hypothesis for a different fraction of the 1000 simulated experiments, but the sizes of the fractions are equivalent and both correspond to the same type-I error rate.