2
$\begingroup$

I have a hypothetical sample of n observations, and I want to test if there are significant differences across multiple treatments.

I need to know how the number of treatments interfere in the type I and type II errors of my analysis

In other words, I need to know if the errors are greater or lower when performing a multiple-comparisons test with n observations and k treatments in relation to the same test performed with n observations and 2k treatments.

$\endgroup$
4
  • $\begingroup$ Welcome to CV. Are you asking how the error rates increase as a function of the number of hypotheses? Have a look here for starters: en.wikipedia.org/wiki/Multiple_comparisons_problem $\endgroup$ Sep 18, 2018 at 22:41
  • $\begingroup$ Thus, as greater is the number of hypotheses, then greater are the errors? $\endgroup$ Sep 18, 2018 at 22:48
  • $\begingroup$ If the hypotheses are independent it won't affect Type I error, however if they dependent then you should correct / adjust the p-values to correct your Type I error. The more hypotheses the more your Type I error increases. $\endgroup$ Sep 19, 2018 at 6:31
  • $\begingroup$ @user2974951 If I flip ten independent coins, each three times, there is a substantially larger chance of getting HHH than if I were to flip only one coin, so independent tests are not invulnerable to the multiple testing problem. $\endgroup$ Sep 19, 2018 at 6:51

1 Answer 1

4
$\begingroup$

In general, the type I error rate increases with the number of independent tests.

If all of the $2k$ treatments you mention are independent, then each of those treatments has its own chance of resulting in a false positive, so the total type I error rate increases.

However, a test can only produce a false positive if a treatment has no effect. In other words, the total type I error rate depends on the number of treatments that actually have an effect. If you assume there no true effects, then every test can only result in a false positive (type I error). This is essentially the justification for the Bonferroni correction, namely:

$$\text{family-wise error rate} = 1 - (1 - \alpha)^k$$ $$p_{\text{Bonferroni}} = p_{\text{original}} \cdot k, \text{ such that:}$$ $$\text{family-wise error rate after adjustment} = 1 - \bigg(1 - \frac{\alpha}{k}\bigg)^k \approx \alpha$$

Where $\alpha$ is the chosen level of significance (e.g. $0.05$) and $k$ is the number of tests.

You can see that if you were to perform twice as many tests without correcting your $p$-values, the type I error rate would inflate considerably. By multiplying the $p$-values by $k$, or (equivalently) dividing $\alpha$ by $k$, the Bonferroni correction controls the family-wise error rate to remain at $\alpha$.

However, as mentioned in the beginning, the actual type I error rate depends on the number of true effects. In fact, if all treatments were to have a reasonably large, clinically relevant effect, then the only thing the Bonferroni correction has accomplished is that it is now a lot harder to pick up those effects! In other words, while the Bonferroni correction controls the type I error rate ($\alpha$), it does so at the cost of an inflated type II error rate ($\beta$).

For this reason, if you have independent tests and you believe it is reasonable to assume that there treatments with a real effect, the Bonferroni correction might be too conservative, which is why there are so many alternatives to it. The most common one is to control the false discovery rate, through the Benjamini-Hochberg procedure:

$$p_{\text{FDR}, i} = p_{\text{original}, i} \cdot \frac{k}{i},$$ $$p_{\text{FDR}, i} = \min(p_{\text{FDR}, i}, \, p_{\text{FDR}, i+1})$$

Where $i$ denotes the rank of the original $p$-values. (See here for detail.)

This procedure penalizes the remaining $p$-values less severely, when the lowest $p$-values are still significant after correction. The justification for this is that if the most significant result is still significant after the Bonferroni penalty, then perhaps not every test has its own false positive (i.e. there are at least some true effects). Hence, this correction might be too harsh and we correct the second most significant $p$-value by a less severe penalty. In doing so, the Benjamini-Hochberg procedure controls the type I error rate, without increasing the type II error rate as much as the Bonferroni correction, which is why it is generally preferred when it is appropriate to use.


Summarized:

  1. Performing multiple tests inflates the type I error rate, as each can yield its own false positive;
  2. Bonferroni correction mitigates this at the cost of power, potentially causing more type II errors;
  3. There are more lenient corrections which try to preserve power, such as Benjamini-Hochberg.
$\endgroup$
1
  • 1
    $\begingroup$ You answered all my questions. I now can do a better work. Thank you, Frans! $\endgroup$ Oct 5, 2018 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.