74
$\begingroup$

I often hear that post hoc tests after an ANOVA can only be used if the ANOVA itself was significant.

  • However, post hoc tests adjust $p$-values to keep the global type I error rate at 5%, don't they?
  • So why do we need the global test first?

  • If we don't need a global test is the terminology "post hoc" correct?

  • Or are there multiple kinds of post hoc tests, some assuming a significant global test result and others without that assumption?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
81
$\begingroup$

Since multiple comparison tests are often called 'post tests', you'd think they logically follow the one-way ANOVA. In fact, this isn't so.

"An unfortunate common practice is to pursue multiple comparisons only when the hull hypothesis of homogeneity is rejected." (Hsu, page 177)

Will the results of post tests be valid if the overall P value for the ANOVA is greater than 0.05?

Surprisingly, the answer is yes. With one exception, post tests are valid even if the overall ANOVA did not find a significant difference among means.

The exception is the first multiple comparison test invented, the protected Fisher Least Significant Difference (LSD) test. The first step of the protected LSD test is to check if the overall ANOVA rejects the null hypothesis of identical means. If it doesn't, individual comparisons should not be made. But this protected LSD test is outmoded, and no longer recommended.

Is it possible to get a 'significant' result from a multiple comparisons test even when the overall ANOVA was not significant?

Yes it is possible. The exception is Scheffe's test. It is intertwined with the overall F test. If the overall ANOVA has a P value greater than 0.05, then the Scheffe's test won't find any significant post tests. In this case, performing post tests following an overall nonsignificant ANOVA is a waste of time but won't lead to invalid conclusions. But other multiple comparison tests can find significant differences (sometimes) even when the overall ANOVA showed no significant differences among groups.

How can I understand the apparent contradiction between an ANOVA saying, in effect, that all group means are identical and a post test finding differences?

The overall one-way ANOVA tests the null hypothesis that all the treatment groups have identical mean values, so any difference you happened to observe is due to random sampling. Each post test tests the null hypothesis that two particular groups have identical means.

The post tests are more focused, so have power to find differences between groups even when the overall ANOVA reports that the differences among the means are not statistically significant.

Are the results of the overall ANOVA useful at all?

ANOVA tests the overall null hypothesis that all the data come from groups that have identical means. If that is your experimental question -- does the data provide convincing evidence that the means are not all identical -- then ANOVA is exactly what you want. More often, your experimental questions are more focused and answered by multiple comparison tests (post tests). In these cases, you can safely ignore the overall ANOVA results and jump right to the post test results.

Note that the multiple comparison calculations all use the mean-square result from the ANOVA table. So even if you don't care about the value of F or the P value, the post tests still require that the ANOVA table be computed.

Improve this answer
$\endgroup$
6
  • 2
    $\begingroup$ This is a great answer Harvey - thanks for writing it! $\endgroup$
    – pmgjones
    Commented Aug 27, 2012 at 23:19
  • 4
    $\begingroup$ (+1) The final two paragraphs provide a good context for understanding and appreciating the entire answer. $\endgroup$
    – whuber ♦
    Commented Jan 16, 2013 at 15:30
  • 8
    $\begingroup$ Excellent answer and I'll add some quotes from Maxwell and Delaney (2004): "...these methods [e.g., Bonferroni, Tukey, Dunnet, etc.] should be viewed as substitutes for the omnibus test because they control alphaEW at thee desired level all by themselves. Requiring a significant omnibus test before proceeding to perform any of these analyses, as is sometimes done, only serves to lower alphaEW below the desired level (Bernhardson, 1975) and hence inappropriately decreases power" (p. 236). $\endgroup$
    – dfife
    Commented Jul 2, 2014 at 12:54
  • 4
    $\begingroup$ While not in the question, I think I should mention - since it may not be obvious - that the reverse situation is also possible in some situations (that an omnibus test rejects but no pairwise comparisons do) $\endgroup$
    – Glen_b
    Commented Oct 4, 2017 at 22:39
  • 3
    $\begingroup$ The lesson in the excellent answer here is to consider carefully what it is you're trying to find out, and then tailor your testing specifically to that end, rather than blindly go "ANOVA test, then test all possible comparisons". $\endgroup$
    – Glen_b
    Commented Jul 21, 2018 at 3:02
33
$\begingroup$

(1) Post hoc tests might or might not achieve the nominal global Type I error rate, depending on (a) whether the analyst is adjusting for the number of tests and (b) to what extent the post-hoc tests are independent of one another. Applying a global test first is pretty solid protection against the risk of (even inadvertently) uncovering spurious "significant" results from post-hoc data snooping.

(2) There is a problem of power. It is well known that a global ANOVA F test can detect a difference of means even in cases where no individual t-test of any of the pairs of means will yield a significant result. In other words, in some cases the data can reveal that the true means likely differ but it cannot identify with sufficient confidence which pairs of means differ.

Improve this answer
$\endgroup$
10
  • $\begingroup$ Re (2): when you say that a one-way ANOVA can report a significant difference when none of the pairwise t-tests does, do you refer to simple non-adjusted ("non post", e.g. not Tukey's procedure or anything) t-tests? I thought this would never be possible, was I wrong? $\endgroup$
    – amoeba
    Commented Jan 22, 2014 at 10:35
  • $\begingroup$ @amoeba That's right; I am referring to unadjusted pairwise tests. Thank you for clarifying this point. $\endgroup$
    – whuber ♦
    Commented Jan 22, 2014 at 16:21
  • 3
    $\begingroup$ @amoba and @whuber: You probably know this, but I’d like to clarify it anyway. Note that it’s possible for the ANOVA test to be significant even if none of the Tukey’s HSD tests are. Simple R example with a balanced data set with three groups: set.seed(249); group = rep(1:3, each=2); y = group + rnorm(6); mod = aov(y~factor(group)); summary(mod); TukeyHSD(mod); plot(y~group) $\endgroup$
    – Karl Ove Hufthammer
    Commented Feb 1, 2014 at 23:15
  • 2
    $\begingroup$ Well, couldn't you at least surmise that there was a difference between the two means with the largest difference between them, since the null hypothesis of the ANOVA is that at least one pair of means differs from each other? $\endgroup$
    – Speldosa
    Commented Feb 27, 2015 at 1:58
  • 2
    $\begingroup$ @Bonferroni If you have a significant global p-value, you have evidence that there is an effect whatsoever. I think this legitimates further analysis. I think the suspect of data snooping is rather natural instead, in case of non-significance of an "omnibus" test. If one person is interested in a specific association, considered as the primary goal of the analysis, then I think such association should be left out from such global test. $\endgroup$
    – Federico Tedeschi
    Commented Jul 12, 2022 at 10:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.