What is the basis for adjusting critical values in multiple t-tests?

Question

Let's say I have three samples: control, A, and B. (I am assuming basic things about sampling like no bias, no confounding factors, and independence in selecting samples.)

I want to see if there is a significant difference between all three samples. I determine significant difference by hypothesis testing using the t-test.

I perform a t-test on each combination of the three samples using the same cutoff $\alpha$. The null hypothesis is that there is no significant difference between the two samples. For sake of argument, in all three tests, all null hypotheses are rejected.

Based on this scenario, I assert:

• The probability of the null hypothesis being rejected in any of the three tests is $3 \cdot \alpha$.
• The probability of the null hypothesis being rejected in all three tests is $\alpha^3$.

Are these assertions correct?

If so, is the basis for doing something like Bonferroni or Holm adjustments of $\alpha$ applies only if the original question is: "Will the null hypothesis be rejected for any of the tests"? If that's the case, are adjustments to $\alpha$ irrelevant if the question is: "Will the null hypothesis be rejected for all of the tests"?

Answer 1

The probability of the null hypothesis being rejected in any of the three tests is 3⋅α.

The probability of any doing a type-I error in at least one of the the three tests is actually:

$1-(1-\alpha)^3$

with a little log() algebra you can get:

$\alpha_{real} = e^{log(1-\alpha_{test})/tests}$ (same as Šidàk's $1-(1-\alpha)^{1/tests}$)

Hope this gives a little clarity on the issue.

UPDATE

Sorry, read your question a little too fast. As I understand it the Holm-Bonferroni method is a stepwise method where you walk through a ranked list of p-values rejecting the null hypothesis as you go, stopping on the first value not fullfilling the criteria. In your case it should be:

1. Lowest p-value $\le \frac{\alpha}{3}$
2. Intermed p-value $\le \frac{\alpha}{2}$
3. Highest p-value $\le \frac{\alpha}{1}$

The chance of all three rejecting the null hypothesis should then be $\frac{\alpha}{3}*\frac{\alpha}{2}*\frac{\alpha}{1}=\frac{\alpha^3}{6}$

The chance of any of the tests being rejected is equal to the first being rejected since you would otherwise not go on testing the next hypothesis, ie $\frac{\alpha}{3}$

The difference in value of a $\alpha/3$ and an $\alpha_{real}$ as I suggested is really small which makes sense since the probability rules should be similar. The Bonferroni method is an old method and I would suggest you use the Šidàk method since it's more precise and if you have a primary outcome you could also try to allocate some probability space to that hypothesis leaving the other to share the rest.

As I understand it the Bonferroni correction is splitting the $\alpha$ for all the tests, this is a cruder method than the Šidàk developed when it was har to calculate $(1-\alpha)^{1/tests}$. If your question is only the Bonferroni method the answer is $\frac{\alpha^3}{9}$ for all three tests and in any of the tests $(1-\frac{\alpha}{3})^3$

If you have all three tests with p-value of $\alpha$ your chance of that should be $\alpha^3$ which probably is $\le\frac{\alpha}{3}$. You also have to reflect about the meaning of it, and I would interpret that as at least one of the tests isn't a type I error, that's not the same as saying that all of them are true.

I also guess your three test hypothesis isn't a-priori and then you probably should do a Bonferroni correction - it sounds to me a little risky to do this post-hoc. The chance that none of the values matches the start-criteria $\le\frac{\alpha}{3}$ seems also somewhat unlikely... I'm not sure I can help you that much more, you can have a look at Holm's original article and see if he mentions your special case.