P-values in multiple comparison t-test

Question

I'm a biologist trying to teach my college class about the issue of multiple comparisons with t-tests.

I've explained that the alpha value must be "combined" so to speak from each of the related tests to determine the effective alpha. Obviously, this results in a higher alpha with each additional related test.

However, what happens with the p-values in multiple comparisons? Would I compare each individual p-value to my effective ("combined") alpha, or would I also consider combining the p-values in some way (perhaps the same way as with the alpha)?

• Ex: if I run 3 t-tests with alpha 0.05, my effective alpha is ~0.14. But let's say I run the tests and the resulting p-values are 0.04, 0.04, and 0.03. Are these 3 p-values taken into account individually (i.e., 0.04 is < 0.14, so it's still significant) or do I combine them together (perhaps as 1 - [(1-0.4) * (1-0.3) * (1-0.4)], which is 0.106).

  • Really, what I'm confused with is that my alpha is "penalized" in multiple comparisons but it seems my p-value is not (or perhaps I've just failed to find a source that discusses this error). Alternatively, if I do try to "penalize" my p-values using the same 1-(1-alpha)^r type of equation, it's still indicating the same significance relationship with my original alpha (as in if my p-val < alpha, it remains < the effective alpha even after adjusting).

  • So knowing the effective alpha is higher than the initial alpha of each test let's me know I'm more likely to commit a Type I error, but it doesn't seem my p-value changes relationship with alpha and so I might still determine it's significant. Does this mean, then, that this is simply a theoretical issue (vs a practical one) and that I would still reject my null hypothesis as I otherwise would, but I'm even more certain that I've committed type I sin?

(I know that corrections (e.g., the Bonfrroni) exist for multiple comparisons, but I'm not trying to "solve" this issue. I'm trying to theoretically (though superficially) explain why multiple comparisons involving t-tests is an issue and why ANOVAs might be employed instead.

Could you explain the relationship of p-values to alpha values in a series of multiple comparison t-tests (done "wrong")?

Answer 1

This is an interesting problem from a pedagogical viewpoint.

In the context of your 3 t-tests example, if you were to use a Bonferonni correction for multiplicity, you would want to perform the first test at a significance level $\alpha_1$, the second test at a significance level $\alpha_2$ and the third test at the significance level $\alpha_3$, so that:

$\alpha_1 + \alpha_2 + \alpha_3 = \alpha$,

where $\alpha$ is the experimentwise error.

You could choose $\alpha_1 = \alpha/3$, $\alpha_2 = \alpha/3$ and $\alpha_3 = \alpha/3$. With this choice, $\alpha$ is equally allocated among the 3 t-tests. (Unequal allocation of $\alpha$ is also possible.)

Let $p_1$ be the unadjusted p-value for the first t-test, $p_2$ be the unadjusted p-value for the second t-test and $p_3$ be the unadjusted p-value for the third t-test.

For each $i$, you can compare $p_i$ against $\alpha_i$ to decide whether or not you can reject the corresponding null hypothesis $H_{0i}$ at the significance level $\alpha_i$. If $p_i \leq \alpha_i$, then $H_{0i}$ can be rejected at the significance level $\alpha_i$. Since $\alpha_i = \alpha/3$, $H_{0i}$ can be rejected when $3 \times p_i \leq \alpha$. The Bonferroni-adjusted p-value for the $i$-th test is therefore $3 \times p_i$.

This paper on Adjusted P-Values for Simultanous Inference by S. Paul Wright (Biometrics 48, 1005-1013, December 1992) will give you a nice overview of other types of p-value adjustments you can make: http://www-stat.wharton.upenn.edu/~steele/Courses/956/Resource/MultipleComparision/Writght92.pdf.