1
$\begingroup$

i created a simple anova with three groups in R and get a significant Result. After that i do a Tukey HSD Test and now nothing is significant. How can that be?

Pr(>F): 0.0316

Group | p adj

1 - 2 | 0.0689014

2 - 3 | 0.0847163

3 - 1 | 0.8909709

$\endgroup$

marked as duplicate by Stefan, Michael Chernick, Jan Kukacka, Bernhard, Ferdi Jun 22 '18 at 11:37

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ thank you for the link. You are right, it's nearly the same. But i still don't know, how i have to evaluate my output. Yes, there is an significantly different between the groups but not between the pairs? Is this good or bad? $\endgroup$ – Robinator Jun 21 '18 at 22:09
1
$\begingroup$

The relationship between the p-values for the F-test and Tukey HSD test is not one-to-one. (even though both test, indirectly, equality of means $\mu_1=\mu_2=\mu_3$)

This is because, for a given distance between the smallest and the largest mean (defining the smallest p-value in Tukey's HSD test), the between group variance (defining the p-value in ANOVA) still depends on the position of the middle mean. The between groups variance is largest when two groups of the three groups cluster together at one end, rather than when the three groups are equally distributed.

For instance: the means 0, 0.5 and 1 have a smaller between groups variance than the means 0, 1 and 1. But the largest distance (between the outside groups) is the same. That means the smallest p-value in Tukey HSD test will not be different for those two cases while the ANOVA p-value does differ.

So for the experiments with the 5% largest significant differences, you do not get the 5% largest F-scores (or vice versa). It depends on the distribution of the groups and two small p-values in the Tukey test just above 5% can make a F-test with a p-value below 5%. (this becomes even stronger when you have a larger number of groups)


The below image is made from a simulation of 1000 draws for three groups of size 50 from a standard normal distribution.

It compares

  • the criteria for the Tukey HSD test (showing the smallest and second smallest p-values on the x- and y-axis with two vertical lines at 0.05 and 0.1)
  • with the criteria for the F-test (the red dots have p-value below 0.05, the green dots have p-value above 0.05 and below 0.1, black dots are above 0.1). comparing Tukey test with F-test

The p-value for the F-test does not align with the smallest p-value for the Tukey HSD test. The p-value for the F-test can be both higher or lower than the p-value for the lowest p-value in the Tukey HSD test, depending on the other p-values in the Tukey HSD test (this is analogous with the earlier mentioned difference between clustered distribution of group means and even distribution of group means).

Note that both the Tukey HSD test (the lowest p-value) and the F-test reject their associated hypothesis for a different fraction of the 1000 simulated experiments, but the sizes of the fractions are equivalent and both correspond to the same type-I error rate.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.