Bonferroni correction: elementary question: when do I reject?

Question

Question practically rewritten as from some answers&comments became clear that I am not understanding the semantics and context.

The "procedure" for Bonferroni is clear to me, my problem is a wrong and/or partial understanding of its semantics and how to use its outputs, so would need help with that.
To make easier to spot what I understand wrongly or miss I make a minimal example of my (mis)understanding and consequente perplexity.

John tests H01 on monday, gets a P-value of .04 and using the traditional alpha = .05 rejects h01.
On tuesday he tests h02, gets P-value .04 and rejects h02.

Bill tests the same two hypothesis together, gets the same p-values as Bill (.04 and .04), applies Bonferroni, alpha/2 = .025, his P values (.04 0.04) are > .025 so he retains h01 and h02 (while John with exactly the same P-values rejected h01 and h02).
This is what seems strange to me.

---

Original Question with some editing
(about testing N times the same hypothesis, which, I learned from answers, is not a correct use of Bonferroni) I think I understand the definition of Bonferroni correction but I do not see written explictly how one should reject the null hypothesis, so that is my naive question: when I perform N experiments, when do I reject the null hypothesis?

• When one (or more) P-values are > alpha/N

• when all P-values > alpha/N

• when more than N/2 P-values > alpha/N

Answer 1

Bonferroni correction is one approach to the problem of multiple testing. Multiple tests mean multiple null hypotheses. So there isn't a single null hypothesis to reject or fail to reject; instead, each $p$-value is an opportunity to reject its corresponding null hypothesis.

Answer 2

Let’s say we have $n=5$ hypothesis tests ($h_0, ..., h_4$) and have decided a sensible threshold to reject the null hypothesis for a single test would be $\alpha=0.05$.

If we wanted to apply a Bonferroni correction to our multiple tests, we would first compute an adjusted $p$-value:

$p_{adj}=\alpha/n=0.05/5=0.01$.

So we have an adjusted $p$-value of 0.01, now let’s say the $p$-values for our 5 tests ($h_0, ..., h_4$) are the following:

($0.0005, 0.8, 0.05, 0.001, 0.01$)

If our criteria for rejecting the null (using our adjusted $p$-value, $p_{adj}=0.01$) is that $p\leq 0.01$, we can then imagine the output of applying this criteria to the above vector would be the following:

($true, false, false, true, true$)

Where $false$ indicates that we fail to reject the null and $true$ that we reject the null and accept the alternate hypothesis.

Scope

To answer the question regarding when to apply corrections, it can be helpful to think of scope, particularly in regards to the data in question.

Let’s say that, for a given analysis, a data set can be used to test several hypotheses. Regardless of chronology, each hypothesis test that makes use of a given data set should should have some sort of adjustment made to the $p$-value. We’re testing the same data multiple times.

Conversely, if we have multiple hypotheses we would like to test on multiple respective data sets that were collected discretely from one another, I don’t believe any adjustment is required.

Of course, if the same random variables were collected for multiple discrete data sets and we can assume the random variables are independent and identically distributed across data sets, we can combine the data sets into a single data set.

Having more data points would increase the power of a given test and give us more confidence in any hypothesis tests that were significant. If we think about it this way, when we apply a Bonferroni correction, it is not dissimilar from randomly splitting a data set into (in the case above) 5 smaller, comparable data sets with an equal number of samples and performing each hypothesis test with an unadjusted $p$-value.

Thus, a Bonferroni correction can be used instead of splitting a data set multiple times because, practically, we may not know how many hypothesis tests will be applied to the data initially (though ideally we would). The Bonferroni correction in that sense is a convenient alternative.