Relation between omnibus test and multiple comparison?

Question

Wikipedia says

Methods which rely on an omnibus test before proceeding to multiple comparisons. Typically these methods require a significant ANOVA/Tukey's range test before proceeding to multiple comparisons. These methods have "weak" control of Type I error.

Also

The F-test in ANOVA is an example of an omnibus test, which tests the overall significance of the model. Significant F test means that among the tested means, at least two of the means are significantly different, but this result doesn't specify exactly what means are different one from the other. Actually, testing means' differences has been made by the quadratic rational F statistic ( F=MSB/MSW). In order to determine which mean differ from another mean or which contrast of means are significantly different, Post Hoc tests (Multiple Comparison tests) or planned tests should be conducted after obtaining a significant omnibus F test. It may be consider using the simple Bonferroni correction or other suitable correction.

So an omnibus test is used to test the overall significance, while multiple comparison is to find which differences are significant.

But if I understand correctly, the main purpose of multiple comparison is to test the overall significance, and it can also find which differences are significant. In other words, multiple comparison can do what an omnibus can do. Then why do we need an omnibus test?

Answer 1

The purpose of of multiple comparisons procedures is not to test the overall significance, but to test individual effects for significance while controlling the experimentwise error rate. It's quite possible for e.g. an omnibus F-test to be significant at a given level while none of the pairwise Tukey tests are—it's discussed here & here.

Consider a very simple example: testing whether two independent normal variates with unit variance both have mean zero, so that

$$H_0: \mu_1=0 \land \mu_2=0$$ $$H_1: \mu_1 \neq 0 \lor \mu_2\neq 0$$

Test #1: reject when $$X_1^2+X_2^2 \geq F^{-1}_{\chi^2_2}(1-\alpha) $$

Test #2: reject when $$|X_1| \lor |X_2|\geq F^{-1}_{\mathcal{N}} \left(1-\frac{1-\sqrt{1-\alpha}}{2}\right)$$

(using the Sidak correction to maintain overall size). Both tests have the same size ($\alpha$) but different rejection regions:

Test #1 is a typical omnibus test: more powerful than Test #2 when both effects are large but neither is so very large. Test #2 is a typical multiple comparisons test: more powerful than Test #1 when either effect is large & the other small, & also enabling independent testing of the individual components of the global null.

So two valid test procedures that control the experimentwise error rate at $\alpha$ are these:

(1) Perform Test #1 & either (a) don't reject the global null, or (b) reject the global null, then (& only in this case) perform Test #2 & either (i) reject neither component, (ii) reject the first component, (ii) reject the second component, or (iv) reject both components.

(2) Perform only Test #2 & either (a) reject neither component (thus failing to reject the global null), (b) reject the first component (thus also rejecting the global null), (c) reject the second component (thus also rejecting the global null), or (d) reject both components (thus also rejecting the global null).

You can't have your cake & eat it by performing Test #1 & not rejecting the global null, yet still going on to perform Test #2: the Type I error rate is greater than $\alpha$ for this procedure.

Answer 2

When testing m hypotheses, there are $2^m$ combinations of hypothesis one can test. One of them is the "global null" hypothesis, a.k.a. the "intersection hypothesis": $\cap H_i^0$.

An omnibus test is typically a name for testing the global null hypothesis. A bare minimum requirement of a multiple testing procedure, is error control under the global null. This is known as "weak FWER" control. But you will probably not stop there- for the purpose of inference on particular hypotheses, you will want a procedure which offer FWER control under any combination of true nulls. This is known as "strong FWER" control.

Answer 3

In addition to the computations associated with Pair-Wise tests, there is something else why ANOVA is used instead of doing all the PAIR-WISE tests.

Sometimes, it is possible that while ANOVA rejects the null hypothesis that all the population means are same at some confidence level, yet if you take all the pair-wise tests (say LSD) you may not find even at least one pair of means which exceeds the difference at that confidence level.

Mathematical proof for the above statement, considering FISHER'S LSD pair-wise tests

here: $S_p$ is the within squares standard deviation.

Take the case, when we have $N$ groups, so, we have $N(N-1)/2$ pair-wise tests.

Sum all such $N(N-1)/2$ tests:

After dividing by $(N-1)$ (as it is the DoF)and squaring on both sides:

on the LHS, we get the same quantity used in ANOVA; However, on the RHS, we get the $N/2$*ANOVA's Test Statistic.

So, even if all the pair-wise LSD tests together cannot reject the null hypotheses, there is still a good chance that ANOVA can reject the null hypotheses.

Hence, ANOVA contains more information than in all the pair wise tests considered together.

P.S: Apologies for using the image instead of typing out the equations.