Supposing we are doing e.g 1000 one-sample t-tests with the null hypothesis that all population means are 0, with significance level 0.01. One common method for dealing with the number of tests is to adjust with the Bonferroni correction so we need each individual t-test to have significance level 0.01/1000 to reject the overall null hypothesis at 0.01 significance

But also under the null hypothesis the number of t-tests that give false positives is binomially distributed with mean 0.01(1000) = 10 and variance 1000(.99)(.1) = 9.9, and this is well-approximated by the Normal distribution given sample size. So can we also just run the t-tests without the Bonferroni correction and use the z-score to decide whether we can reject the null-hypothesis at 0.01 significance?

To me these seem like they should be equivalent ways to get at the same answer and they seem to use the same assumptions, but I’m curious if there are any edge-cases/situations where these methods might disagree?


I don't understand what you mean in the second method. For each of the 1000 t-tests, you reject the null hypothesis if the p-value is less than 0.01. Then you count how many tests were rejected. You expect to have about 10 if the null hypothesis is true and the variance of the number is about 9.9. So, let's suppose you observe 10 rejected null hypotheses. Then, what do you do? I would think that is about what I would expect even if all null hypotheses were true. So, I would just assume they are all true. But, what if you observe 20 rejected null hypotheses?

The Bonferroni adjustment makes no assumptions about the joint distribution of the test statistics. The probability of rejecting one or more true null hypotheses is less than or equal to alpha regardless of how many of them are true and regardless of the joint distribution of the test statistics.

Although I don't understand your second method, you are assuming the test statistics are independent when you state the number of rejected null hypotheses has a binomial distribution.

Another difference between the two is that the first method tells you which hypotheses can be rejected, it is not just a statement about the overall truth of all 1000 null hypotheses. Your second method seems to be to first count how many are rejected, then use that number to draw some inference about whether all 1000 are true.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.