I'm a biologist trying to teach my college class about the issue of multiple comparisons with t-tests.

I've explained that the alpha value must be "combined" so to speak from each of the related tests to determine the effective alpha. Obviously, this results in a higher alpha with each additional related test.

However, what happens with the p-values in multiple comparisons? Would I compare each individual p-value to my effective ("combined") alpha, or would I also consider combining the p-values in some way (perhaps the same way as with the alpha)?

  • Ex: if I run 3 t-tests with alpha 0.05, my effective alpha is ~0.14. But let's say I run the tests and the resulting p-values are 0.04, 0.04, and 0.03. Are these 3 p-values taken into account individually (i.e., 0.04 is < 0.14, so it's still significant) or do I combine them together (perhaps as 1 - [(1-0.4) * (1-0.3) * (1-0.4)], which is 0.106).

    • Really, what I'm confused with is that my alpha is "penalized" in multiple comparisons but it seems my p-value is not (or perhaps I've just failed to find a source that discusses this error). Alternatively, if I do try to "penalize" my p-values using the same 1-(1-alpha)r type of equation, it's still indicating the same significance relationship with my original alpha (as in if my p-val < alpha, it remains < the effective alpha even after adjusting).

    • So knowing the effective alpha is higher than the initial alpha of each test let's me know I'm more likely to commit a Type I error, but it doesn't seem my p-value changes relationship with alpha and so I might still determine it's significant. Does this mean, then, that this is simply a theoretical issue (vs a practical one) and that I would still reject my null hypothesis as I otherwise would, but I'm even more certain that I've committed type I sin?

(I know that corrections (e.g., the Bonfrroni) exist for multiple comparisons, but I'm not trying to "solve" this issue. I'm trying to theoretically (though superficially) explain why multiple comparisons involving t-tests is an issue and why ANOVAs might be employed instead.

Could you explain the relationship of p-values to alpha values in a series of multiple comparison t-tests (done "wrong")?

  • $\begingroup$ I've been scouring my old textbooks and the internet for a few days, but I can only seem to find results about the effects on alpha or on corrections like the Bonferroni. However, I'm just trying to figure out how p-values themselves actually can be wrongly calculated in multiple comparisons (especially since it seems that simply adjusting the alpha doesn't make it any harder to have a p-value lower than my target alpha value)... $\endgroup$ Aug 30 '19 at 14:17
  • 1
    $\begingroup$ In the first bullet, comparing each p-value to the original alpha (0.05) gets you an effective alpha of ~0.14. That is, by calling each one significant at a 5% type I error rate, you get an overall type I error rate of 14%. At no point should you be directly comparing individual p-values to the effective alpha of 0.14 - that results in rather permissive tests with an even higher effective alpha! P value adjustment allows you to revise that individual-level alpha downward to get an overall alpha that matches your target type I error rate of 5%. $\endgroup$ Aug 30 '19 at 15:04

This is an interesting problem from a pedagogical viewpoint.

In the context of your 3 t-tests example, if you were to use a Bonferonni correction for multiplicity, you would want to perform the first test at a significance level $\alpha_1$, the second test at a significance level $\alpha_2$ and the third test at the significance level $\alpha_3$, so that:

$\alpha_1 + \alpha_2 + \alpha_3 = \alpha$,

where $\alpha$ is the experimentwise error.

You could choose $\alpha_1 = \alpha/3$, $\alpha_2 = \alpha/3$ and $\alpha_3 = \alpha/3$. With this choice, $\alpha$ is equally allocated among the 3 t-tests. (Unequal allocation of $\alpha$ is also possible.)

Let $p_1$ be the unadjusted p-value for the first t-test, $p_2$ be the unadjusted p-value for the second t-test and $p_3$ be the unadjusted p-value for the third t-test.

For each $i$, you can compare $p_i$ against $\alpha_i$ to decide whether or not you can reject the corresponding null hypothesis $H_{0i}$ at the significance level $\alpha_i$. If $p_i \leq \alpha_i$, then $H_{0i}$ can be rejected at the significance level $\alpha_i$. Since $\alpha_i = \alpha/3$, $H_{0i}$ can be rejected when $3 \times p_i \leq \alpha$. The Bonferroni-adjusted p-value for the $i$-th test is therefore $3 \times p_i$.

This paper on Adjusted P-Values for Simultanous Inference by S. Paul Wright (Biometrics 48, 1005-1013, December 1992) will give you a nice overview of other types of p-value adjustments you can make: http://www-stat.wharton.upenn.edu/~steele/Courses/956/Resource/MultipleComparision/Writght92.pdf.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.