Why is a "Correction" Required in Multiple Hypothesis Testing?

Question

Suppose I have the following dataset that contains measurements on different patients (each unique ID is a unique patient) - "x" variables represent covariates and "y" variables represent outcomes:

 id       x_1       x_2       x_3        x_4       x_5       y_1       y_2        y_3       y_4       y_5       y_6       y_7        y_8        y_9      y_10
1  1 137.71008  83.22509  96.77519 139.199703 -31.05143  20.78144 201.37399 251.993953 343.69970 231.62894 278.26058  63.90400 179.571790  -7.370581 135.52294
2  2 240.57995  76.04897 -21.00893   3.407949 161.11235 115.08062 101.07260 200.581716 112.08013 177.05726 215.56689 145.16217 109.613811 112.834290  37.73935
3  3  66.92601 179.04881 -34.18888   3.106041 225.28584  56.88339 220.26244   3.416566 172.49610  54.01245  11.05028  53.05884 100.335158  77.712706 296.31327
4  4 120.87087 202.61092 176.98454 140.890102 142.51842 -64.06773 -23.43913 169.147210 129.27064 -70.55632  22.39471  30.87697 147.580290  43.288333 111.33670
5  5  56.29673  48.33402 119.25437  45.504666  89.87237 198.60063 181.48716  24.956806  55.75533 101.08906  18.19344 163.20589   5.623714  85.519361 148.23663
6  6  60.13320 156.73718 -10.77539 -79.319398 -47.54688 127.28152  49.17664 126.881145  37.45836 142.68339 103.83295 231.97907 186.168837 235.994598 230.42569

I am interested in fitting the following regression models - I believe this is called "Multiple Regression":

model_1 <- lm(y_1 ~ x_1 + x_2  + x_3  + x_4  + x_5 , data = my_data)
model_2 <- lm(y_2 ~ x_1 + x_2  + x_3  + x_4  + x_5 , data = my_data)
model_3 <- lm(y_3 ~ x_1 + x_2  + x_3  + x_4  + x_5 , data = my_data)
model_4 <- lm(y_4 ~ x_1 + x_2  + x_3  + x_4  + x_5 , data = my_data)
model_5 <- lm(y_5 ~ x_1 + x_2  + x_3  + x_4  + x_5 , data = my_data)
model_6 <- lm(y_6 ~ x_1 + x_2  + x_3  + x_4  + x_5 , data = my_data)
model_7 <- lm(y_7 ~ x_1 + x_2  + x_3  + x_4  + x_5 , data = my_data)
model_8 <- lm(y_8 ~ x_1 + x_2  + x_3  + x_4  + x_5 , data = my_data)
model_9 <- lm(y_9 ~ x_1 + x_2  + x_3  + x_4  + x_5 , data = my_data)
model_10 <- lm(y_10 ~ x_1 + x_2  + x_3  + x_4  + x_5 , data = my_data)

Based on the results of these regression models, suppose I want to see that if each "beta" coefficient associated with "x_1" for each model can be considered statistically significant or not.

I have been told that for such problems, a "correction" such as the Bonferroni Correction is often required to choose a more "conservative" value of "alpha" that will be used in these hypothesis tests. But I struggle to understand why this necessary. For example, if I were to inspect the results of these models:

> model_1

Call:
lm(formula = y_1 ~ x_1 + x_2 + x_3 + x_4 + x_5, data = my_data)

Coefficients:
(Intercept)          x_1          x_2          x_3          x_4          x_5  
  131.15352     -0.09218     -0.03841      0.02782     -0.07882     -0.05099  

> model_7

Call:
lm(formula = y_7 ~ x_1 + x_2 + x_3 + x_4 + x_5, data = my_data)

Coefficients:
(Intercept)          x_1          x_2          x_3          x_4          x_5  
  108.07151     -0.06348      0.01031     -0.05734     -0.06381     -0.12770  

I can't seem to understand why determining if "-0.09218" is statistically significant - how this somehow is related to whether "-0.06348" is statistically significant. The calculations required to calculate the beta coefficients for "model 1" do not seem to directly affect the calculations required to calculate the beta coefficients for "model 7" - even though both of these models are using the same dataset. This being said, why should testing the statistical significance of coefficients from "model 1" require somehow "correcting" for "model 2" and vice versa?

Can someone please help me understand as to why in this problem that I have outlined, a "correction factor" is required for individual comparisons and why all individual hypothesis tests for these models need to be compared at a lower value of "alpha" - even though all these individual comparisons seem independent?

I can understand if someone was interested in determining if all "x_1" coefficients for all models were JOINTLY statistically significant why a correction factor might be needed - but I am having trouble understand why testing whether " -0.09218" is statistically significant by itself still required a correction factor. Where exactly is the "Family" in this case?

Thanks!

Answer 1

This is a tricky topic: when exactly do you correct for multiple testing? The two extremes are both problematic:

• never correcting for multiple testing will result in too many false positives,
• always correcting for multiple testing seems impossible, e.g. if, over your carreer as a statistician you perform $1\,000$ (which is still a conservative estimate) tests you wouldn't use $\alpha = \frac{5\%}{1\,000}$ for each of those tests.

At the end of the day you will end up somewhere in the middle: you will account for multiple testing "in batches" and you'll have to decide on how to "batch" tests (or confidence intervals for that matter!) together.

As a frequentist (which I assume you are because you are interested here in NHST) you are interested not in the result of a single test (that will be correct or wrong, but you won't know which of the two scenarios you are in) but rather in properties of your procedure if it were performed repeatedly. Now what "the procedure" is depends on the context.

One strategy is to do this on a paper-by-paper basis, i.e. each paper gets a budget of $\alpha = 5\%$ that you can spend. Still, if sample sizes are low (as they usually are) and effect sizes are comparably small (as they usually are) and you are interested in many things at once (as one usually is), correcting for every test will be unsatisfactory. Then one has to make a decision: What is the primary interest of this analysis / paper? For these tests / confidence intervals you correct for multiple testing and are thus allowed to do a "hard" interpretation of the results. All other analyses are declared secondary analyses and the interpretation of results is more exploratory, e.g. generating hypothesis for follow-up studies.

Similarly you can "batch" tests / confidence intervals together if you want to interpret the results of these analyses together: "If $\beta_1$ in this model is X and $\beta_1$ in the second model is Y then Z." Note that this also implies that you do not have to account for having $\beta_2, \dots, \beta_5$ in your models - if you do not want to interpret these in the end that is.

All of this assumes of course that you have decided on a testing strategy before looking at the data.

Answer 2

There is a lot of arbitrariness in certain statistical practices. As other responses and previous discussions in the literature point out, it's very hard to justify the tradition that you correct for multiple comparisons that you do within one study/one paper. This is a convention that has developed, but honestly has a somewhat shaky basis.

On the other hand, it is clear that we want to avoid that the scientific literature gets flooded with false research "findings" (although arguably that is already a problem):

• We know that if we do lots of comparisons, then even if there is just nothing going on about 5% will end up having $p\leq 0.05$.
• Of course, in practice we investigate a mixture of effects that are there or not, that are small or large etc., so we don't know whether we're in that situation.
• People are sometimes surprised to learn that all else being equal $p\leq 0.05$ is less likely to be a genuine effect in the direction indicated by the point estimate, if the study was less well powered. I.e. "The study was small, but we nevertheless achieved significance, so the effect must be especially strong as also indicated by the large estimate!" is exactly the wrong way around. This is particularly obvious if you think about the "valid" frequentist study design of collecting no data and rejecting the null hypothesis randomly with probability 5%. In that case, it's clear that the p-value is completely useless, because the study is so underpowered.
• Finally, the current $p\leq 0.05$ "standard" is arguably already a rather weak standard for "statistical significance", so findings supported by the typical evidence behind $p\leq 0.05$ are not really that credible in the first place (i.e. even without multiplicity issues).
• What I'm trying to get at here, is that if you really want to do null hypothesis testing and, thus, presumably also care about the familywise type I error rate, you really ought to be worried about multiple comparisons to some extent.
• You can, of course, also read all of the above as a criticism of p-values and of over-emphasizing "significance" (see also here, here and here).

So, given how weak $p\leq 0.05$ is and how this gets worse in small studies, you do not really want to pile multiple comparisons on top of that. Otherwise, the false positive rate of your work will get pretty high pretty fast. On that basis, one should not ignore multiplicity and should consider it a problem to the reliability of the scientific literature. Of course, if one considers the purpose of "science" to be for "scientists" to be able to make more "findings", publish them and as a result get tenure, one might not consider that a problem.

Exactly how one deals with multiplicity is a different question. E.g. whether you should care about the familywise type I error rate, or perhaps rather about the false discovery rate, or perhaps about the probability that claims are true given all available information (or alternatively "if one were a bit skeptical of new claims"), is open for debate. Within any of these frameworks, there's then many different methods for doing what you set out to do (e.g. the Bonferroni correction, which as pointed out by others is unnecessarily conservative).

Especially within the type of exploratory kind of work being described in the original question, where there is a bit of an fishing expedition for "signals", I do not think that a confirmatory study mindset is quite right (i.e. where you have a clearly defined hypothesis and test it with a study) and something more oriented towards minimizing false positive findings from exploratory work would seem like a better fit (where p-values may be the wrong tool in the first place, although one can try to correct them in ways that aim to control e.g. the false discovery rate). It's just important to then not misleadingly report such work as if it had been been from confirmatory work.

Answer 3

I'm frequently asked when multiple comparison adjustment should be used. Then I start talking about false discovery rate, type-I errors and so on, to conclude that I was not fully understood. So I came with a following example (maybe "show don't tell" rule applies to data analysis too ;) ):

Imagine, you compare three parameters, P1, P2 and P3 in two groups (control and treatment, say), and get p=0,03 for each parameter. Now, if you apply Bonferroni correction, you'll reduce number of significant results from 3 to 0 (or to 1 or 2 with other corrections). But you really, really love your tripple significance, so you write three articles, one for each parameter, and send them to three distinct journals. Now, no one asks you for multiple comparisons adjustment! The moral is: if you can sensibly (forget about ethics for a while) split your results into distinct papers/threads, you probably do not need adjustments.