I'm a biologist trying to teach my college class about the issue of multiple comparisons with t-tests.
I've explained that the alpha value must be "combined" so to speak from each of the related tests to determine the effective alpha. Obviously, this results in a higher alpha with each additional related test.
However, what happens with the p-values in multiple comparisons? Would I compare each individual p-value to my effective ("combined") alpha, or would I also consider combining the p-values in some way (perhaps the same way as with the alpha)?
Ex: if I run 3 t-tests with alpha 0.05, my effective alpha is ~0.14. But let's say I run the tests and the resulting p-values are 0.04, 0.04, and 0.03. Are these 3 p-values taken into account individually (i.e., 0.04 is < 0.14, so it's still significant) or do I combine them together (perhaps as
1 - [(1-0.4) * (1-0.3) * (1-0.4)], which is 0.106).
Really, what I'm confused with is that my alpha is "penalized" in multiple comparisons but it seems my p-value is not (or perhaps I've just failed to find a source that discusses this error). Alternatively, if I do try to "penalize" my p-values using the same
1-(1-alpha)r type of equation, it's still indicating the same significance relationship with my original alpha (as in if my p-val < alpha, it remains < the effective alpha even after adjusting).
So knowing the effective alpha is higher than the initial alpha of each test let's me know I'm more likely to commit a Type I error, but it doesn't seem my p-value changes relationship with alpha and so I might still determine it's significant. Does this mean, then, that this is simply a theoretical issue (vs a practical one) and that I would still reject my null hypothesis as I otherwise would, but I'm even more certain that I've committed type I sin?
(I know that corrections (e.g., the Bonfrroni) exist for multiple comparisons, but I'm not trying to "solve" this issue. I'm trying to theoretically (though superficially) explain why multiple comparisons involving t-tests is an issue and why ANOVAs might be employed instead.
Could you explain the relationship of p-values to alpha values in a series of multiple comparison t-tests (done "wrong")?