13
$\begingroup$

Intro: Having noted the attention received today by this question, "Can ANOVA be significant when none of the pairwise t-tests is?," I thought I might be able to reframe it in an interesting way that would deserve its own set of answers.

A variety of incongruous results (at face value) can occur when statistical significance is understood as a simple dichotomy and judged on the mere basis of which is higher, the $p$ or the $\alpha$. @Glen_b's answer to the above question presents a useful example of a case where:

  • An ANOVA $F$-test produces a $p_F<.05$ for one independent variable (IV) with four levels, but
  • $p_t>.08$ for all two-sample $t$-tests that compare differences in the same dependent variable (DV) among observations corresponding to each pair of the IV's four levels.

A similar case arose despite Bonferroni corrections for post-hoc pairwise comparisons via this question: Anova repeated measures is significant, but all the multiple comparisons with Bonferroni correction are not? Previously mentioned cases with a slightly different test in multiple regression also exist:

I bet that in cases like these, some (but not all) pairwise comparisons' (or regression coefficients' significance tests') $p$ values must be fairly close to $\alpha$ if a corresponding omnibus test can achieve a $p <\alpha$. I see this is the case in @Glen_b's first example, where $F_{(3,20)}=3.19$, $p_F=.046$, and the largest pairwise difference gives the smallest $p_t=.054$. Must this be the case in general? More specifically:


Question: If an ANOVA $F$-test produces a $p_F=.05$ for one polytomous IV's effect on a continuous DV, how high could the lowest $p$ value be among all two-sample $t$-tests that compare each pair of the IV's levels? Could the minimum pairwise significance be as high as $p_t=.50$?


I welcome answers that address only this specific question. However, to further motivate this question, I'll elaborate and throw in some potentially rhetorical questions. Feel welcome to address these concerns as well, and even to ignore the specific question if you like, especially if the specific question gets a definitive answer.

Significance: Consider how much less important the difference between a $p_F=.04$ and a $p_t=.06$ would be if statistical significance were judged in continuous terms of the strength of evidence against the null hypothesis (Ron Fisher's approach, I think?), rather than in dichotomous terms as above or below an $\alpha=.05$ threshold for acceptable probability of error in choosing whether to reject the null wholesale. "$p$-hacking" is a known problem that partly owes its notoriety to an unnecessary vulnerability introduced by interpretation of $p$ values according to the common practice of dichotomizing significance into the equivalents of "good enough" and "not good enough." If one were to dispose this practice and focus instead on interpreting $p$ values as strength of evidence against the null on a continuous interval, might omnibus testing be somewhat less important when one really cares about multiple pairwise comparisons? Not useless necessarily, as any reasonably efficient improvement in statistical accuracy is of course desirable, but...if, for instance, the lowest pairwise comparison's $p$ value is necessarily within $.10$ of the ANOVA (or other omnibus test) $p$ value, doesn't this make the omnibus test somewhat more trivial, less compulsory, and even more misleading (in conjunction with preexisting misunderstandings), especially if one doesn't particularly want to control $\alpha$ across multiple tests?

Conversely, if data may exist such that an omnibus $p=.05$, but all pairwise $p>.50$, shouldn't this further motivate omnibus and contrast testing throughout practice and pedagogy? It seems to me that this issue should also inform the relative merits of judging statistical significance according to a dichotomy vs. a continuum, in that the dichotomous interpretive system should be more sensitive to small adjustments when differences are "marginally significant", whereas neither system is safe from a failure to perform an omnibus test or adjust for multiple comparisons if this difference / adjustment can be very large (e.g., $p_t-p_F>.40)$ in theory.

Other optional complexities to consider or ignore—whatever makes answering easier and more worthwhile:

$\endgroup$
  • 1
    $\begingroup$ You might want to clarify whether the pairwise t-tests should use the same error variance estimate as the omnibus F-test (in Glen's example they don't). $\endgroup$ – Scortchi Jan 23 '14 at 16:30
  • 1
    $\begingroup$ I meant an ordinary t-test for the difference in means using $t=(\bar{y}_1-\bar{y}_2)/\left({\hat\sigma\sqrt{\frac{1}{n_1} + \frac{1}{n_2}}}\right)$, but with $\hat\sigma$ calculated as the square root of the ANOVAR mean square error. It is the usual post-hoc pairwise t-test & doesn't adjust for multiple comparisons, unlike Tukey's HSD. It does incorporate information from all groups, but is independent of differences in group means. $\endgroup$ – Scortchi Jan 23 '14 at 17:01
  • 1
    $\begingroup$ I see (sort of)! I'd be primarily interested in following @Glen_b's example and not using $\sqrt{\text{MSE}}$, but using the first formula you mentioned to avoid incorporating info from all groups. That's not to say I have a strong preference here...but part of my original intention was to present a variant of the common theme in these questions: "What's the real harm in ignoring information beyond the two particular groups in question for any given two-sample test among many?" I guess that theme is worth carrying through in this decision as well. $\endgroup$ – Nick Stauner Jan 23 '14 at 17:14
  • 1
    $\begingroup$ @Scortchi I have included an example at the other question which covers your first comment (i.e. where the tests are done using the common error variance and df), though all the tests (F and multiple comparisons) are done at a pretty low significance level (0.0025, not 0.05). When compared to the individual ordinary two-sample t-tests as is being asked by Nick S. here, it shows that quite a substantial difference in significance is possible (in this case, $p_t>.05$ for all the ordinary t-tests, yet $p_F<0.002$). I believe with many groups, it's possible to go much further. $\endgroup$ – Glen_b Jan 24 '14 at 0:56
  • 1
    $\begingroup$ I sketched an answer to the first part of this question a few minutes ago in a comment at stats.stackexchange.com/questions/83030/…. $\endgroup$ – whuber Jan 24 '14 at 17:09
8
$\begingroup$

Assuming equal $n$s [but see note 2 below] for each treatment in a one-way layout, and that the pooled SD from all the groups is used in the $t$ tests (as is done in usual post hoc comparisons), the maximum possible $p$ value for a $t$ test is $2\Phi(-\sqrt{2}) \approx .1573$ (here, $\Phi$ denotes the $N(0,1)$ cdf). Thus, no $p_t$ can be as high as $0.5$. Interestingly (and rather bizarrely), the $.1573$ bound holds not just for $p_F=.05$, but for any significance level we require for $F$.

The justification is as follows: For a given range of sample means, $\max_{i,j}|\bar y_i - \bar y_j| = 2a$, the largest possible $F$ statistic is achieved when half the $\bar y_i$ are at one extreme and the other half are at the other. This represents the case where $F$ looks the most significant given that two means differ by at most $2a$.

So, without loss of generality, suppose that $\bar y_.=0$ so that $\bar y_i=\pm a$ in this boundary case. And again, without loss of generality, suppose that $MS_E=1$, as we can always rescale the data to this value. Now consider $k$ means (where $k$ is even for simplicity [but see note 1 below]), we have $F=\frac{\sum n\bar y^2/(k-1)}{MS_E}= \frac{kna^2}{k-1}$. Setting $p_F=\alpha$ so that $F=F_\alpha=F_{\alpha,k-1,k(n-1)}$, we obtain $a =\sqrt{\frac{(k-1)F_\alpha}{kn}}$. When all the $\bar y_i$ are $\pm a$ (and still $MS_E=1$), each nonzero $t$ statistic is thus $t=\frac{2a}{1\sqrt{2/n}} = \sqrt{\frac{2(k-1)F_\alpha}{k}}$. This is the smallest maximum $t$ value possible when $F=F_\alpha$.

So you can just try different cases of $k$ and $n$, compute $t$, and its associated $p_t$. But notice that for given $k$, $F_\alpha$ is decreasing in $n$ [but see note 3 below]; moreover, as $n\rightarrow\infty$, $(k-1)F_{\alpha,k-1,k(n-1)} \rightarrow \chi^2_{\alpha,k-1}$; so $t \ge t_{min} =\sqrt{2\chi^2_{\alpha,k-1}/k}$. Note that $\chi^2/k=\frac{k-1}k \chi^2/(k-1)$ has mean $\frac{k-1}k$ and SD$\frac{k-1}k\cdot\sqrt{\frac2{k-1}}$. So $\lim_{k\rightarrow\infty}t_{min} = \sqrt{2}$, regardless of $\alpha$, and the result I stated in the first paragraph above is obtained from asymptotic normality.

It takes a long time to reach that limit, though. Here are the results (computed using R) for various values of $k$, using $\alpha=.05$:

k       t_min    max p_t   [ Really I mean min(max|t|) and max(min p_t)) ]
2       1.960     .0500
4       1.977     .0481   <--  note < .05 !
10      1.840     .0658
100     1.570     .1164
1000    1.465     .1428
10000   1.431     .1526

A few loose ends...

  1. When k is odd: The maximum $F$ statistic still occurs when the $\bar y_i$ are all $\pm a$; however, we will have one more at one end of the range than the other, making the mean $\pm a/k$, and you can show that the factor $k$ in the $F$ statistic is replaced by $k-\frac 1k$. This also replaces the denominator of $t$, making it slightly larger and hence decreasing $p_t$.
  2. Unequal $n$s: The maximum $F$ is still achieved with the $\bar y_i = \pm a$, with the signs arranged to balance the sample sizes as nearly equally as possible. Then the $F$ statistic for the same total sample size $N = \sum n_i$ will be the same or smaller than it is for balanced data. Moreover, the maximum $t$ statistic will be larger because it will be the one with the largest $n_i$. So we can't obtain larger $p_t$ values by looking at unbalanced cases.
  3. A slight correction: I was so focused on trying to find the minimum $t$ that I overlooked the fact that we are trying to maximize $p_t$, and it is less obvious that a larger $t$ with fewer df won't be less significant than a smaller one with more df. However, I verified that this is the case by computing the values for $n=2,3,4,\ldots$ until the df are high enough to make little difference. For the case $\alpha=.05, k\ge 3$ I did not see any cases where the $p_t$ values did not increase with $n$. Note that the $df=k(n-1)$ so the possible df are $k,2k,3k,\ldots$ which get large fast when $k$ is large. So I'm still on safe ground with the claim above. I also tested $\alpha=.25$, and the only case I observed where the $.1573$ threshold was exceeded was $k=3,n=2$.
$\endgroup$
  • 1
    $\begingroup$ Russ, this is a nice answer. One small question: in your second paragraph where you have $\max|\bar y_j - \bar y_j| = 2a$, did you intend $\max|\bar y_i - \bar y_j| = 2a$? $\endgroup$ – Glen_b Mar 1 '15 at 5:53
  • $\begingroup$ Good catch. I corrected it. $\endgroup$ – rvl Mar 2 '15 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.