# Incremental adjustments of $\alpha$ for multiple hypothesis testing?

Given a nominal significance level of $\alpha$ for a single hypothesis test, I understand that if you do multiple tests, $i = 1$ to $m$, then according to Dunn's Bonferroni correction, the significance level $\alpha(m)$ of the last test $m$ should be reduced to $\alpha/m$.

Now my question is, what should the significance level be for the preceding tests, $i = 1$ to $m-1$?

According to Bonferroni, $\alpha(i) = \alpha/i$ is applied to the p-value from each successive test - or is it $\alpha(i) = \alpha/m$ for all tests?

But according to Benjamini-Hochberg (1995), $\alpha(i) = \alpha*i/m$ is effectively applied to the ranked p-values once all the results are in.

But these approaches are apparently inconsistent, as Bonferroni gives a curved progression of $\alpha(i)$ to $\alpha(m)$, while Benjamini-Hochberg (1995) gives a straight line progression.

So which is the correct approach? And what if you don't even know how many tests $m$ you will end up doing? Or what if you think you know, but decide anyway to add more tests later? How should $\alpha(i)$ change with each successive test?

In particular, if you apply Benjamini-Hochberg's linear progression, and decide to add more tests later, does that mean you have to go back and recalibrate the significance of all of your earlier tests? And if that is the case, is it statistically correct to take a decision on whether or not to perform additional tests, given that you can already calculate how that will impact the significance of your earlier tests?

• This question somehow confuses the order in which the experiments are performed and p-values are computed (you talk about "preceding" and "succesive" tests and about "incremental" adjustments) and the ranked order of the p-values that one needs for the BH procedure. Certainly the p-values from any actual sequence of experiments will not be ordered! – amoeba says Reinstate Monica Apr 11 '16 at 12:50
• Yes, hence the question whether the p-values from earlier experiments would then have to be re-ranked as more experiments are added. – Kelvin Apr 11 '16 at 13:36

The Bonferroni correction tests each individual hypothesis at the level of $\alpha/m$, where $m$ is the number of tests. There is no difference in this limit based on the order of doing the tests, as your question seems to suggest. This is an attempt to control the family-wise error rate (FWER). At FWER of 0.05, you expect to make some false-positive error in one out of 20 repetitions of the entire sampling and hypothesis-testing process. There are less conservative ways to control FWER, as the pages linked here indicate.

Benjamini and Hochberg propose a way to control something different, the false discovery rate (FDR). That's the expected fraction of total positive calls that are false-positives. That's a different goal than controlling the FWER, with control of the FDR intended to miss fewer true-positives than FWER.

Thinking about the difference between FWER and FDR can be difficult at first, but it's worth the effort.

If you have a small number of pre-specified hypotheses to test, then there may need to be no multiple-correction test at all. Control for FWER or FDR is typically used in cases where all reasonable hypotheses are being tested at once; for example, which of 20,000 genes is more highly expressed in some experimental than in a control condition.

Safest approach would be to consider all the hypotheses you want to test and testing them together with FWER or FDR control. Doing a few tests first and then devising more tests on the same data based on the results of the initial tests runs a risk of data dredging. In your significance testing you would then need somehow to take into account that the generation of the subsequent hypotheses depended on finding the initial hypotheses to be true, not an easy thing for which to correct.

Congratulations, you are learning that these issues are not simple and there is not a single answer that covers everything.

You need to be careful in interpreting significant results when the researcher has unlimited degrees of freedom in the tests that they can do. There are arguments that you should adjust for the potential number of tests that could have been done, not just the number that actually were done.

If you keep adding tests, then eventually you will find some analysis that is significant, even if there is no real relationship. This is easy to see by using your favorite statistical package to generate 20 columns of random data with no planned relationship (generate each column independently of the others), now compute all the pairwise correlations and test if they are equal to 0. The truth is that the population or process correlations are all 0, but it would be very unlikely to not find at least one significant when looking at each test. So the adjustment idea is to take into account the number of tests. Now consider, what if you don't do all the tests, but first do a scatterplot matrix (look at all the pairwise scatterplots) and find the pair that looks to have the strongest relationship and only test that one pair. Now I have only done one actual test, but since I could have done all the others, the most accurate result would still adjust for all the tests that I could have done, but did not do.

A realistic concern is that if I am the manufacturer of Drug A and want to compare it to drug B, so I do a large randomized trial where patients are randomized to drug and at the end I see no significant difference between the 2 groups, but I still want to sell my drug, so I start looking at subgroups, is drug A more effective in Males? Females?, Older Patients? Younger Patients? Red Haired Patients? Green Eyed Patients? etc. If I analyze enough subgroups, I will eventually find something significant by chance alone, akin to the random example above. If I look at plots of the data, or other summaries, I can reduce the number of actual tests (and therefore how many I adjust for).

On the other hand, if I am now looking at side effects and my initial test shows that there are more deaths in the group receiving drug A than in the group receiving drug B, with a p-value of 0.02 (assuming traditional alpha of 0.05), but I don't want to share that. So I start adding some more tests and find that the rate of drowsiness is not significantly different between the 2 groups and the rate of rashes is not significantly different. So now I combine all 3 tests and adjust for multiple comparisons and report no significant differences in side effects now. Here the adjustment has benefited me as an unethical advertiser.

One approach to deal with both of the above problems is to preregister all planned analyses and to not do anything that was not planned ahead of time. More realistically we state the planned comparisons and report on those, but then also do exploratory analyses beyond those, but those results are dubbed "Exploratory" and used to generate new studies, not as final answers.

But what if death was not one of my planned outcomes, but the data safety monitoring board points out that there are significantly fewer deaths in the treatment group than the control group? This would be an exploratory outcome, but do we have the equipoise to start a new study looking at death as the outcome and randomize subjects to control?

You can see a lot more discussion on some of these ideas by doing an internet search (google or other) on "Andrew Gelman garden of forking paths".