# P-values in multiple comparison t-test

I'm a biologist trying to teach my college class about the issue of multiple comparisons with t-tests.

I've explained that the alpha value must be "combined" so to speak from each of the related tests to determine the effective alpha. Obviously, this results in a higher alpha with each additional related test.

However, what happens with the p-values in multiple comparisons? Would I compare each individual p-value to my effective ("combined") alpha, or would I also consider combining the p-values in some way (perhaps the same way as with the alpha)?

• Ex: if I run 3 t-tests with alpha 0.05, my effective alpha is ~0.14. But let's say I run the tests and the resulting p-values are 0.04, 0.04, and 0.03. Are these 3 p-values taken into account individually (i.e., 0.04 is < 0.14, so it's still significant) or do I combine them together (perhaps as 1 - [(1-0.4) * (1-0.3) * (1-0.4)], which is 0.106).

• Really, what I'm confused with is that my alpha is "penalized" in multiple comparisons but it seems my p-value is not (or perhaps I've just failed to find a source that discusses this error). Alternatively, if I do try to "penalize" my p-values using the same 1-(1-alpha)r type of equation, it's still indicating the same significance relationship with my original alpha (as in if my p-val < alpha, it remains < the effective alpha even after adjusting).

• So knowing the effective alpha is higher than the initial alpha of each test let's me know I'm more likely to commit a Type I error, but it doesn't seem my p-value changes relationship with alpha and so I might still determine it's significant. Does this mean, then, that this is simply a theoretical issue (vs a practical one) and that I would still reject my null hypothesis as I otherwise would, but I'm even more certain that I've committed type I sin?

(I know that corrections (e.g., the Bonfrroni) exist for multiple comparisons, but I'm not trying to "solve" this issue. I'm trying to theoretically (though superficially) explain why multiple comparisons involving t-tests is an issue and why ANOVAs might be employed instead.

Could you explain the relationship of p-values to alpha values in a series of multiple comparison t-tests (done "wrong")?

• I've been scouring my old textbooks and the internet for a few days, but I can only seem to find results about the effects on alpha or on corrections like the Bonferroni. However, I'm just trying to figure out how p-values themselves actually can be wrongly calculated in multiple comparisons (especially since it seems that simply adjusting the alpha doesn't make it any harder to have a p-value lower than my target alpha value)... – theforestecologist Aug 30 '19 at 14:17
• In the first bullet, comparing each p-value to the original alpha (0.05) gets you an effective alpha of ~0.14. That is, by calling each one significant at a 5% type I error rate, you get an overall type I error rate of 14%. At no point should you be directly comparing individual p-values to the effective alpha of 0.14 - that results in rather permissive tests with an even higher effective alpha! P value adjustment allows you to revise that individual-level alpha downward to get an overall alpha that matches your target type I error rate of 5%. – Nuclear Wang Aug 30 '19 at 15:04

This is an interesting problem from a pedagogical viewpoint.

In the context of your 3 t-tests example, if you were to use a Bonferonni correction for multiplicity, you would want to perform the first test at a significance level $$\alpha_1$$, the second test at a significance level $$\alpha_2$$ and the third test at the significance level $$\alpha_3$$, so that:

$$\alpha_1 + \alpha_2 + \alpha_3 = \alpha$$,

where $$\alpha$$ is the experimentwise error.

You could choose $$\alpha_1 = \alpha/3$$, $$\alpha_2 = \alpha/3$$ and $$\alpha_3 = \alpha/3$$. With this choice, $$\alpha$$ is equally allocated among the 3 t-tests. (Unequal allocation of $$\alpha$$ is also possible.)

Let $$p_1$$ be the unadjusted p-value for the first t-test, $$p_2$$ be the unadjusted p-value for the second t-test and $$p_3$$ be the unadjusted p-value for the third t-test.

For each $$i$$, you can compare $$p_i$$ against $$\alpha_i$$ to decide whether or not you can reject the corresponding null hypothesis $$H_{0i}$$ at the significance level $$\alpha_i$$. If $$p_i \leq \alpha_i$$, then $$H_{0i}$$ can be rejected at the significance level $$\alpha_i$$. Since $$\alpha_i = \alpha/3$$, $$H_{0i}$$ can be rejected when $$3 \times p_i \leq \alpha$$. The Bonferroni-adjusted p-value for the $$i$$-th test is therefore $$3 \times p_i$$.

This paper on Adjusted P-Values for Simultanous Inference by S. Paul Wright (Biometrics 48, 1005-1013, December 1992) will give you a nice overview of other types of p-value adjustments you can make: http://www-stat.wharton.upenn.edu/~steele/Courses/956/Resource/MultipleComparision/Writght92.pdf.