How much smaller can $p$ values from ANOVA's $F$-test be vs. those from multiple $t$-tests on the same data?

Question

Intro: Having noted the attention received today by this question, "Can ANOVA be significant when none of the pairwise t-tests is?," I thought I might be able to reframe it in an interesting way that would deserve its own set of answers.

A variety of incongruous results (at face value) can occur when statistical significance is understood as a simple dichotomy and judged on the mere basis of which is higher, the $p$ or the $\alpha$. @Glen_b's answer to the above question presents a useful example of a case where:

• An ANOVA $F$-test produces a $p_F<.05$ for one independent variable (IV) with four levels, but
• $p_t>.08$ for all two-sample $t$-tests that compare differences in the same dependent variable (DV) among observations corresponding to each pair of the IV's four levels.

A similar case arose despite Bonferroni corrections for post-hoc pairwise comparisons via this question: Anova repeated measures is significant, but all the multiple comparisons with Bonferroni correction are not? Previously mentioned cases with a slightly different test in multiple regression also exist:

• Why is it possible to get significant F statistic (p<.001) but non-significant regressor t-tests?: $p_F<.001,p_{\beta t}>.09$
• How can a regression be significant yet all predictors be non-significant?
  • In @whuber's answer, $p_F=.0003,p_{\beta t}>.09$

I bet that in cases like these, some (but not all) pairwise comparisons' (or regression coefficients' significance tests') $p$ values must be fairly close to $\alpha$ if a corresponding omnibus test can achieve a $p <\alpha$. I see this is the case in @Glen_b's first example, where $F_{(3,20)}=3.19$, $p_F=.046$, and the largest pairwise difference gives the smallest $p_t=.054$. Must this be the case in general? More specifically:

---

Question: If an ANOVA $F$-test produces a $p_F=.05$ for one polytomous IV's effect on a continuous DV, how high could the lowest $p$ value be among all two-sample $t$-tests that compare each pair of the IV's levels? Could the minimum pairwise significance be as high as $p_t=.50$?

---

_{I welcome answers that address only this specific question. However, to further motivate this question, I'll elaborate and throw in some potentially rhetorical questions. Feel welcome to address these concerns as well, and even to ignore the specific question if you like, especially if the specific question gets a definitive answer.}

Significance: Consider how much less important the difference between a $p_F=.04$ and a $p_t=.06$ would be if statistical significance were judged in continuous terms of the strength of evidence against the null hypothesis (Ron Fisher's approach, I think?), rather than in dichotomous terms as above or below an $\alpha=.05$ threshold for acceptable probability of error in choosing whether to reject the null wholesale. "$p$-hacking" is a known problem that partly owes its notoriety to an unnecessary vulnerability introduced by interpretation of $p$ values according to the common practice of dichotomizing significance into the equivalents of "good enough" and "not good enough." If one were to dispose this practice and focus instead on interpreting $p$ values as strength of evidence against the null on a continuous interval, might omnibus testing be somewhat less important when one really cares about multiple pairwise comparisons? Not useless necessarily, as any reasonably efficient improvement in statistical accuracy is of course desirable, but...if, for instance, the lowest pairwise comparison's $p$ value is necessarily within $.10$ of the ANOVA (or other omnibus test) $p$ value, doesn't this make the omnibus test somewhat more trivial, less compulsory, and even more misleading (in conjunction with preexisting misunderstandings), especially if one doesn't particularly want to control $\alpha$ across multiple tests?

Conversely, if data may exist such that an omnibus $p=.05$, but all pairwise $p>.50$, shouldn't this further motivate omnibus and contrast testing throughout practice and pedagogy? It seems to me that this issue should also inform the relative merits of judging statistical significance according to a dichotomy vs. a continuum, in that the dichotomous interpretive system should be more sensitive to small adjustments when differences are "marginally significant", whereas neither system is safe from a failure to perform an omnibus test or adjust for multiple comparisons if this difference / adjustment can be very large (e.g., $p_t-p_F>.40)$ in theory.

_{Other optional complexities to consider or ignore—whatever makes answering easier and more worthwhile:}

• ^{How high $p$s for $t$s could be if, for $F$, $p<.05$ instead (e.g., $p=.01, .001,\dots$)}
• ^{Sensitivity to number of levels in a polytomous IV}
• ^{Sensitivity to unevenness in significance of pairwise differences (while all $p_t>p_F$)}
  • ^{whuber's answer indicates that including small differences can mask big differences.}
• ^{Differences among various omnibus tests' corrections for multiple comparisons}
  • ^{See also: Correcting for multiple comparisons in a within subjects / repeated measures ANOVA; excessively conservative?}
  • ^{With multiple IVs, it seems multicollinearity can exacerbate this issue.}
• ^{Restricted cases where data meet all assumptions of classic parametric tests optimally}
  • ^{This restriction may be important to prevent this question from being somewhat moot.}

Answer 1

Assuming equal $n$s [but see note 2 below] for each treatment in a one-way layout, and that the pooled SD from all the groups is used in the $t$ tests (as is done in usual post hoc comparisons), the maximum possible $p$ value for a $t$ test is $2\Phi(-\sqrt{2}) \approx .1573$ (here, $\Phi$ denotes the $N(0,1)$ cdf). Thus, no $p_t$ can be as high as $0.5$. Interestingly (and rather bizarrely), the $.1573$ bound holds not just for $p_F=.05$, but for any significance level we require for $F$.

The justification is as follows: For a given range of sample means, $\max_{i,j}|\bar y_i - \bar y_j| = 2a$, the largest possible $F$ statistic is achieved when half the $\bar y_i$ are at one extreme and the other half are at the other. This represents the case where $F$ looks the most significant given that two means differ by at most $2a$.

So, without loss of generality, suppose that $\bar y_.=0$ so that $\bar y_i=\pm a$ in this boundary case. And again, without loss of generality, suppose that $MS_E=1$, as we can always rescale the data to this value. Now consider $k$ means (where $k$ is even for simplicity [but see note 1 below]), we have $F=\frac{\sum n\bar y^2/(k-1)}{MS_E}= \frac{kna^2}{k-1}$. Setting $p_F=\alpha$ so that $F=F_\alpha=F_{\alpha,k-1,k(n-1)}$, we obtain $a =\sqrt{\frac{(k-1)F_\alpha}{kn}}$. When all the $\bar y_i$ are $\pm a$ (and still $MS_E=1$), each nonzero $t$ statistic is thus $t=\frac{2a}{1\sqrt{2/n}} = \sqrt{\frac{2(k-1)F_\alpha}{k}}$. This is the smallest maximum $t$ value possible when $F=F_\alpha$.

So you can just try different cases of $k$ and $n$, compute $t$, and its associated $p_t$. But notice that for given $k$, $F_\alpha$ is decreasing in $n$ [but see note 3 below]; moreover, as $n\rightarrow\infty$, $(k-1)F_{\alpha,k-1,k(n-1)} \rightarrow \chi^2_{\alpha,k-1}$; so $t \ge t_{min} =\sqrt{2\chi^2_{\alpha,k-1}/k}$. Note that $\chi^2/k=\frac{k-1}k \chi^2/(k-1)$ has mean $\frac{k-1}k$ and SD$\frac{k-1}k\cdot\sqrt{\frac2{k-1}}$. So $\lim_{k\rightarrow\infty}t_{min} = \sqrt{2}$, regardless of $\alpha$, and the result I stated in the first paragraph above is obtained from asymptotic normality.

It takes a long time to reach that limit, though. Here are the results (computed using R) for various values of $k$, using $\alpha=.05$:

k       t_min    max p_t   [ Really I mean min(max|t|) and max(min p_t)) ]
2       1.960     .0500
4       1.977     .0481   <--  note < .05 !
10      1.840     .0658
100     1.570     .1164
1000    1.465     .1428
10000   1.431     .1526

---

A few loose ends...

1. When k is odd: The maximum $F$ statistic still occurs when the $\bar y_i$ are all $\pm a$; however, we will have one more at one end of the range than the other, making the mean $\pm a/k$, and you can show that the factor $k$ in the $F$ statistic is replaced by $k-\frac 1k$. This also replaces the denominator of $t$, making it slightly larger and hence decreasing $p_t$.
2. Unequal $n$s: The maximum $F$ is still achieved with the $\bar y_i = \pm a$, with the signs arranged to balance the sample sizes as nearly equally as possible. Then the $F$ statistic for the same total sample size $N = \sum n_i$ will be the same or smaller than it is for balanced data. Moreover, the maximum $t$ statistic will be larger because it will be the one with the largest $n_i$. So we can't obtain larger $p_t$ values by looking at unbalanced cases.
3. A slight correction: I was so focused on trying to find the minimum $t$ that I overlooked the fact that we are trying to maximize $p_t$, and it is less obvious that a larger $t$ with fewer df won't be less significant than a smaller one with more df. However, I verified that this is the case by computing the values for $n=2,3,4,\ldots$ until the df are high enough to make little difference. For the case $\alpha=.05, k\ge 3$ I did not see any cases where the $p_t$ values did not increase with $n$. Note that the $df=k(n-1)$ so the possible df are $k,2k,3k,\ldots$ which get large fast when $k$ is large. So I'm still on safe ground with the claim above. I also tested $\alpha=.25$, and the only case I observed where the $.1573$ threshold was exceeded was $k=3,n=2$.