3
$\begingroup$

From answers on this site I have learned that it is possible to get significant results from multiple comparison tests ("post hoc"), even if ANOVA did not yield a significant result. I believe that the opposite is also possible – significant ANOVA result, and non-significant post-hoc results.

I am not sure that it is possible to answer a general question, but I would still like to ask it – which is generally speaking, more powerful, an ANOVA test or post hoc tests on the same data?

$\endgroup$
12
  • 6
    $\begingroup$ They test different hypotheses. $\endgroup$
    – AdamO
    Apr 1, 2021 at 15:32
  • 1
    $\begingroup$ I agree, but I think that nevertheless one can still ask which is more powerful. $\endgroup$
    – Sam
    Apr 4, 2021 at 8:31
  • 1
    $\begingroup$ @Sam No, you cannot meaningfully ask which is more powerful, since the definition of powerful is $1 - P(\text{Reject H}_{0} | \text{H}_{0} = \text{False})$. The "$|\text{H}_{0} = \text{False}$" part means the ANOVA power ≠ the post hoc pairwise test power in the same way that apple ≠ orange. $\endgroup$
    – Alexis
    Apr 6, 2021 at 0:34
  • $\begingroup$ @Alexis The question most certainly is meaningful, indeed it is quite standard. My answer details exactly how the comparison is made and how the posthoc t-tests test (amongst other things) the same null hypothesis as the F-test. $\endgroup$ Apr 6, 2021 at 1:32
  • 1
    $\begingroup$ @GordonSmyth Your rather belabored answer (I downvoted) mixes the issues of Reject ANOVA but Not Reject any post hoc tests and vice versa (a question which has been asked and answered repeatedly on this site, BTW) with some generalized notion of which test is "more powerful". ANOVA and post hoc tests have different null and alternative hypotheses. Since power is literally and explicitly defined with respect to a specific null hypothesis and set of alternative hypotheses, you are comparing apples to oranges. $\endgroup$
    – Alexis
    Apr 6, 2021 at 15:21

1 Answer 1

7
$\begingroup$

To make anova and posthoc tests comparable, you need to be conducting posthoc tests for at least $G-1$ contrasts (where $G$ is the number of groups) such that the contrasts span the space of all possible contrasts. In that case, accepting all the contrast null hypotheses implies that the true group means are all equal, equivalent to the anova F-test null hypothesis.

Given this assumption, which is more powerful depends on the configuration of true group means and on the specific post-hoc tests you plan to conduct.

Let's say you are doing a oneway anova with $G$ groups. While there are many ways to conduct post hoc tests, the following strategies are amongst the most common choices:

  1. Perform an overall F-test for differences between the group means.

  2. Perform t-tests for all possible pairwise comparisons. Adjust the p-values for multiple testing using Holm's method.

  3. Choose $G-1$ linearly independent contrasts that correspond to your scientific hypotheses. Conduct a t-test for each contrast and adjust the p-values using Holm's method.

Consider the overall null hypothesis that the true group means are all equal. In approach 1, the null hypothesis is rejected if the F-test p-value is less than $\alpha$, where $\alpha$ is the significance level. In approaches 2 or 3, the null hypothesis of no differences is rejected if any of the adjusted p-values are less than $\alpha$. All three approaches control the type I error rate for this test at $\alpha$.

Approaches 2 and 3 test a number of null hypotheses besides the overall null of no differences, but I am answering your question in terms of the overall null. For approaches 2 and 3, the minimum adjusted p-value is an effective test of the overall null because the intersection of the null hypotheses for the individual t-tests is equal to the overall null hypothesis and the union of t-test alternative hypotheses is equal to the F-test alternative.

The F-statistic can be written as a weighted average of the squared t-statistics from approaches 2 or 3. Hence it works best when all or most of the t-tests contribute meaningfully to the average. For the t-test approaches, the result is driven mainly or entirely by the largest t-statistic. In general, the F-statistic will give a smaller p-value if the individual t-statistics are all similar in size whereas approach 3 will give a smaller p-value if one of the t-statistics is much larger than the others in absolute size.

The F-test is more powerful than the t-tests if the true group means are equally spaced. The contrasts t-tests will be more powerful if one or more of the contrasts match the true pattern of group differences. The pairwise t-tests are generally less powerful than the contrast t-tests (because more $G(G-1)/2$ tests are conducted instead of $G-1$) but still may be more powerful than the F-test if one of the pairwise differences is much larger than the others.

One way to choose the contrasts is to test each group mean vs the average of the other group means. This approach is more powerful than the other approaches for detecting one group separate to the others.

After many years of working in biomedical research, I find that I use the F-test less and less and I use approach 3 more and more. The main trouble with the F-test in practice is that, when the null hypothesis is rejected, it gives no guidance as to which groups means are different. So one has to conduct the t-tests anyway, in which case disagreement between the t-tests and the F-tests becomes an interpretation problem. Assuming that the contrasts are appropriately chosen, approach 3 is statistically more powerful as well as simpler to interpret. Approaches 2 and 3 have the added advantage of the F-test of strong familywise error rate control over all tests conducted.

If the anova is balanced (equal numbers in each group) then one can also consider Tukey's honestly significant differences, which is similar to making all possible pairwise comparisons but more powerful because Tukey's method accounts for the dependencies between the pairwise comparisons. My remarks above apply to any anova, balanced or otherwise.

$\endgroup$
1
  • $\begingroup$ Balance is not required. Procedures that account for dependencies exactly in unbalanced situations, for any set of contrasts, have been standard for years; see eg the 2010 book "Multiple Comparisons Using R" and references therein. $\endgroup$ Apr 6, 2021 at 1:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.