I read the following paper: Perneger (1998) What's wrong with Bonferroni adjustments.

The author summarized by saying that Bonferroni adjustment have, at best, limited applications in biomedical research and should not be used when assessing evidence about specific hypothesis:

Summary points:

  • Adjusting statistical significance for the number of tests that have been performed on study data—the Bonferroni method—creates more problems than it solves
  • The Bonferroni method is concerned with the general null hypothesis (that all null hypotheses are true simultaneously), which is rarely of interest or use to researchers
  • The main weakness is that the interpretation of a finding depends on the number of other tests performed
  • The likelihood of type II errors is also increased, so that truly important differences are deemed non-significant
  • Simply describing what tests of significance have been performed, and why, is generally the best way of dealing with multiple comparisons

I have the following data set and I want to do multiple testing correction BUT I am unable to decide for the best method in this case.

enter image description here

I want to know if it is imperative to do this kind of correction for all the data sets that contain lists of means and what is the best method for the correction in this case?

  • $\begingroup$ what exactly is 'mean A', 'meanB' ... ? $\endgroup$ – user83346 Aug 5 '16 at 5:03
  • 5
    $\begingroup$ By not correcting for multiple comparisons you run the risk of irreproducible results. Many fields, including medicine and psychology, have recently discovered that's exactly what has happened: much of what they "know" based on uncorrected p-values turns out just not to be so. Without meaning to seem cynical, it looks like the choice is clear: the researcher who needs to meet a p-value criterion to publish will not correct; the sceptic who wants knowledge will. $\endgroup$ – whuber Aug 13 '16 at 17:26
  • $\begingroup$ @whuber but can it be still considered reproducible when so many different methods to correct for p-values are available? In his answer martino even gives guidelines to choose between less conservative or more powerful methods. $\endgroup$ – Nakx Feb 14 '18 at 4:54
  • 3
    $\begingroup$ @Nakx Reproducibility is only loosely associated with the statistical procedure: it refers to whether or not comparable results will be obtained when the research is independently done by others (and presumably in such attempts to replicate, a single clear hypothesis will be articulated in advance and a statistical procedure appropriate to that hypothesis will be used). If the original procedure does not produce a correct p-value, then when used many times for many independent studies it will on average make more irreproducible determinations than its users intend or expect. $\endgroup$ – whuber Feb 14 '18 at 14:03

What is wrong with the Bonferroni correction besides the conservatism mentioned by others is what's wrong with all multiplicity corrections. They do not follow from basic statistical principles and are arbitrary; there is no unique solution to the multiplicity problem in the frequentist world. Secondly, multiplicity adjustments are based on the underlying philosophy that the veracity of one statement depends on which other hypotheses are entertained. This is equivalent to a Bayesian setup where the prior distribution for a parameter of interest keeps getting more conservative as other parameters are considered. This does not seem to be coherent. One could say that this approach comes from researchers having been "burned" by a history of false positive experiments and now they want to make up for their misdeeds.

To expand a bit, consider the following situation. An oncology researcher has made a career of studying efficacy of chemotherapies of a certain class. All previous 20 of her randomized trials have resulted in statistically insignificant efficacy. Now she is testing a new chemotherapy in the same class. The survival benefit is significant with $P=0.04$. A colleague points out that there was a second endpoint studied (tumor shrinkage) and that a multiplicity adjustment needs to be applied to the survival result, making for an insignificant survival benefit. How is it that the colleague emphasized the second endpoint but couldn't care less about adjusting for the 20 previous failed attempts to find an effective drug? And how would you take into account prior knowledge about the 20 previous studies if you weren't Bayesian? What if there had been no second endpoint. Would the colleague believe that a survival benefit had been demonstrated, ignoring all previous knowledge?

  • 2
    $\begingroup$ Not clear on the reference to 'repeatable'. If there is a single test, with no multiplicity adjustment required, the chance that a result with $P=0.04$ is repeated is not high. $\endgroup$ – Frank Harrell Oct 17 '14 at 17:10
  • 2
    $\begingroup$ To answer @MJA I think there are two preferred approaches: (1) be Bayesian or (2) prioritize the hypotheses and report the results in context, in priority order. $\endgroup$ – Frank Harrell Oct 17 '14 at 17:11
  • 4
    $\begingroup$ There is nothing principled about that nor is it exact in any way. Bonferroni's inequality is an upper bound for the error probability only. Why spend $\alpha$ equally on 5 parameters? Why not make an ellipsoidal region instead of a rectangular one for the acceptance region? Why not use Scheffe or Tukey's method? Why not use a simple composite ANOVA-type test? You do not achieve the desired $\alpha$ by using an inequality. $\endgroup$ – Frank Harrell Aug 4 '16 at 21:18
  • 3
    $\begingroup$ You are equivocating two error rates. Under the null, Bonferroni EXACTLY maintains the expected number of errors per family. It gives an UPPER BOUND on the probability of "at least one" error per family (which depends on correlation). Spending alpha equally on the 5 tests is perfectly logical given no particular reason to prioritize the tests in a different way. Given another context, there are principled reasons to do otherwise. You seem to imply that it's "unprincipled" to use a mathematically sound method simply because alternative methods exist given other contexts, goals and assumptions. $\endgroup$ – Bonferroni Aug 4 '16 at 21:34
  • 3
    $\begingroup$ @FrankHarrell Your other questions only serve to illustrate my point. There are often numerous choices of test statistic, test procedure, etc., even in the absence of multiplicity. That doesn't make the methodology "arbitrary" in the sense you seem to be implying. If one is interested in an omnibus test, then by all means conduct one. If one is only interested in the univariate tests, then by all means conduct the univariate tests. Are you seriously suggesting that it's "arbitrary" to select the test that addresses the question you're interested in rather than some other question? $\endgroup$ – Bonferroni Aug 4 '16 at 21:43

He summarized saying that Bonferroni adjustment have, at best, limited applications in biomedical research and should not be used when assessing evidence about specific hypothesis.

The Bonferroni correction is one of the simplest and most conservative multiple comparisons technique. It is also one of the oldest and has been improved upon greatly over time. It is fair to say that the Bonferroni adjustments have limited application in almost all situations. There is almost certainly a better approach. That is to say, you will need to correct for multiple comparisons but you can choose a method that is less conservative and more powerful.

Less Conservative

Multiple comparisons methods protect against getting at least one false positive in a family of tests. If you perform one test at the $\alpha$ level then you are allowing a 5% chance of getting a false positive. In other words, you reject your null hypothesis erroneously. If you perform 10 tests at the $\alpha = 0.05$ level then this increases to $1-(1-0.05)^{10}$ = ~40% chance of getting a false positive

With the Bonferroni method you use an $\alpha_b$ at the lowest end of the scale (i.e. $\alpha_b = \alpha/n$) to protect your family of $n$ tests at the $\alpha$ level. In other words, it is the most conservative. Now, you can increase $\alpha_b$ above the lower limit set by Bonferroni (i.e. make your test less conservative) and still protect your family of tests at the $\alpha$ level. There are many ways to do this, the Holm-Bonferroni method for example or better still False Discovery Rate

More Powerful

A good point brought up in the paper referenced is that the likelihood of type II errors is also increased so that truly important differences are deemed non-significant.

This is very important. A powerful test is one that finds significant results if they exist. By using the Bonferroni correction you end up with a less powerful test. As Bonferroni is conservative, the power is likely to be considerable reduced. Again, one of the alternative methods eg False Discovery Rate, will increase the power of the test. In other words, not only do you protect against false positives, you also improve your ability to find truly significant results.

So yes, you should apply some correction technique when you have multiple comparisons. And yes, Bonferroni should probably be avoided in favour of a less conservative and more powerful method

  • $\begingroup$ There are several alternatives – Holm Bonferroni for example is simple and easy to understand. Why not give it a go. Let’s say you application is in gene expression or protein expression where you are testing possibly thousands of variables in an experiment then you FDR is typically used. $\endgroup$ – martino Oct 17 '14 at 13:08
  • $\begingroup$ Your method of calculating the 40% chance of false positive in ten tests is premised on your tests being independent events but with real data this is quite unlikely to be the case. I think that is at least worthy of comment. $\endgroup$ – Silverfish Aug 5 '16 at 10:00
  • $\begingroup$ I'm also concerned this answer seems to conflate methods of preserving familywise error rate with those for false discovery rate. It isn't a bad idea to be discussing both these things, but since they do different jobs I don't think they should be presented as equivalent $\endgroup$ – Silverfish Aug 5 '16 at 10:02
  • $\begingroup$ But if I understand well, the FDR (false discovery rates) do not guarantee type I error control at a predetermined level ? (see also my answer to this question) $\endgroup$ – user83346 Aug 5 '16 at 10:51
  • $\begingroup$ But isn't it more transparent and useful to report all the raw p-values in an article, so that readers can judge by themselves of their validity or choose which of the myriad of adjustment methods they want to use? $\endgroup$ – Nakx Feb 14 '18 at 4:51

Thomas Perneger is not a statistician and his paper is full of mistakes. So I wouldn't take it too seriously. It's actually been heavily criticized by others. For example, Aickin said Perneger's paper "consists almost entirely of errors": Aickin, "Other method for adjustment of multiple testing exists", BMJ. 1999 Jan 9; 318(7176): 127.

Also, none of the p-values in the original question are < .05 anyway, even without multiplicity adjustment. So it probably doesn't matter what adjustment (if any) is used.

  • 4
    $\begingroup$ Thanks for the link! I've added a fuller reference. This is still more of a comment than an answer & I'm sure you've something of interest to add, or at least a brief summary of what Aicken says. Unrelated to that: to say Perneger has no expertise in statistics doesn't seem true (by any reasonable standard), amiable, or useful - would you consider removing the statement? $\endgroup$ – Scortchi - Reinstate Monica Aug 5 '16 at 10:53
  • $\begingroup$ @Scortchi I've changed "has no expertise in statistics" to "is not a statistician." Incidentally, I disagree that it's not useful to distinguish expert opinions from non-expert opinions. $\endgroup$ – Bonferroni Aug 5 '16 at 23:51
  • 2
    $\begingroup$ As far as I can tell, Perneger has no degree in statistics and has never published a paper in a statistical journal. The paper cited in the question is an opinion article in BMJ that's been called out for being completely wrong. So what is Perneger's supposed expertise that's indisputable "beyond any reasonable standard?" Being "amiable" shouldn't get in the way of the truth. $\endgroup$ – Bonferroni Aug 7 '16 at 15:12
  • 3
    $\begingroup$ As far as I can tell he's a professor at a university hospital with a Masters in Biostatistics & a PhD in Epidemiology who lectures in Medical Statistics & publishes analyses of clinical trials & observational studies in medical journals. If you deduce from that "no statistical expertise", I think your standard's rather higher than you might reasonably expect your readers to assume. (Which is what I should've said rather than that the standard was unreasonable.) Anyway, thanks for editing it! $\endgroup$ – Scortchi - Reinstate Monica Aug 7 '16 at 20:18

Maybe it's good to explain the ''reasoning behind'' multiple testing corrections like the one of Bonferroni. If that is clear then you will be able to judge yourself whether you should apply them or not.

In a hypothesis test one tries to find evidence for some known or assumed fact about the real world. It is similar to ''proof by contradiction'' in mathematics, i.e. if one wants to prove that e.g. a parameter $\mu$ is non-zero, then one will assume that the opposite is true, i.e. one assumes that $H_0: \mu=0$ and one tries to find something that is impossible under that assumption. In statistics things are rarely impossible, but they can be very improbable.

So if we want to show that $H_1: \mu \ne 0$ then we assume the opposite namely $H_0: \mu = 0$ and we try to find something very improbable. Very improbable is defined in terms of a probability lower than an a priori fixed significance level $\alpha$. Note that, because of the analogy I will use terms such as ''statistically proven'' or ''statistical evidence'', these terms aree just used for didactical reasons and are not used in general.

In order to find that ''low probability'' we draw a random sample form a distribution that is known when $H_0$ (our assumption of the ''opposite'' of what we want to prove) is true. As we assumted $H_0$ te be true we can compute the probability of this outcome (more precise something that is at least as extreme as this outcome).

As the sample is a random draw from a distribution, it may be that we obtain a low probability just by ''bad luck with the sample'' and then we reject $H_0$ just because we had bad luck with the sample. Rejecting $H_0$ means that we consider to have found evidence for $H_1$ but it is false evidence in these cases where we have bad luck with the sample.

False evidence is a bad thing in science because we believe to have gained true knowledge about the world, but in fact we may have had bad luck with the sample. This kinds of errors should consequently be controled. Therefore one should put an upper limit on the probability of this kind of evidence, or one should control the type I error. This is done by fixing an acceptable significance level in advance.

So if we fix our significance level at $5\%$ then we are saying that we are ready to reject $H_0$ when it is true (because of bad luck with the sample) with a chance of $5\%$. As (see supra) rejecting $H_0$ is ''statistical evidence'' for $H_1$ this means that we falsely consider $H_1$ as ''statistically proven''.

Assume now that we have two parameters, and we want to show that that at least one is different from zero. Follwing the logic of ''proof by contradiction'' we will assume $H_0: \mu_1=0 \& \mu_2=0$ versus $H_1: \mu1 \ne 0 | \mu_2 \ne 0$ and that we use a signficance level $\alpha=0.05$.

One possibility to do this is to split this hypothesis test and to test $H_0^{(1)}: \mu_1=0$ versus $H_0^{(1)}: \mu_1 \ne 0$ and to test $H_1^{(2)}: \mu_2=0$ versus $H_1^{(2)}: \mu_2 \ne 0$ both at the significance level $\alpha=0.05$.

To do both tests we draw one sample , so we use one and the same sample to do both of these tests. I may have bad luck with that one sample and erroneously reject $H_0^{(1)}$ but with that same sample I may also have bad luck with the sample for the second test and erroneously reject $H_0^{(1)}$

Therefore, the chance that at least one of the two is an erroneous rejection is 1 minus the probability that both are not rejected, i.e. $1-(1-0.05)^2=0.0975$, where it was assumed that both tests are independent. In other words, the type I error has ''inflated'' to 0.0975 which is almost double $\alpha$.

The important fact here is that the two tests are based on one and the sampe sample !

Note that we have assumed independence. If you can not assume independence then you can show, using the Bonferroni inequality$ that the type I error can inflate up to 0.1.

Note that Bonferroni is conservative and that Holm's stepwise procedure holds under the same assumptions as for Bonferroni, but Holm's procedure has more power.

When the variables are discrete it's better to use test statistics based on the minimum p-value and if you are ready to abandon type I error control when doing a massive number of tests then False Discovery Rate procedures may be more powerful.


If e.g. (see the example in the answer by @Frank Harrell)

$H_0^{(1)}: \mu_1=0$ versus $H_1^{(1)}: \mu_1 \ne 0$ is the a test for the effect of a chemotherapy and

$H_0^{(2)}: \mu_1=0$ versus $H_1^{(2)}: \mu_2 \ne 0$ is the test for the effect on tumor shrinkage,

then, in order to control the type I error at 5% for the hypothesis $H_0^{(12)}: \mu_1=0 \& \mu_2 = 0$ versus $H_1^{(12)}: \mu_1 \ne 0 | \mu_2 \ne 0$ (i.e. the test that at least one of them has an effect) can be carried out by testing (on the same sample)

$H_0^{(1)}$ versus $H_1^{(1)}$ at the 2.5% level and also $H_0^{(2)}$ versus $H_1^{(2)}$ at the 2.5% level.

  • 2
    $\begingroup$ I think this question benefits from an answer like this but I suggest tightening the wording of "So if we fix our significance level at 5% then we are saying that we are ready to accept to find false evidence (because of bad luck with the sample) with a chance of 5%"... That is only the probability of error if the null is actually true, and that's worth saying. (Is "false evidence" a common term? I'm more used to seeing "false positive".) $\endgroup$ – Silverfish Aug 5 '16 at 10:10
  • $\begingroup$ @Silverfish; I re-phresed it a bit, do you think it is better like this ? $\endgroup$ – user83346 Aug 5 '16 at 10:17
  • 1
    $\begingroup$ I think that's better - "statistically proven" would probably benefit from rephrasing too, I know this is how many people interpret p<0.05 or whatever but of course it isn't really a proof! $\endgroup$ – Silverfish Aug 5 '16 at 10:24
  • $\begingroup$ @Silverfish: I fully agree that is not a ''proof'' but I used the term for didactial reasons, because I started by the analogy to proof by contradiction. I will add such a clarification at the begining $\endgroup$ – user83346 Aug 5 '16 at 10:46
  • $\begingroup$ Your Edit is confusing. "The effect of chemotherapy" in Frank's example is measured via two measures: survival rate and tumor shrinkage. Both can be influenced by chemo. The hypothesis is obviously that chemo works. But "works" can be quantified in two different ways. That's the vagueness aspect I've been talking about in your new thread. $\endgroup$ – amoeba Aug 13 '16 at 14:10

A nice discussion of Bonferroni correction and effect size http://beheco.oxfordjournals.org/content/15/6/1044.full.pdf+html Also, Dunn-Sidak correction and Fisher's combined probabilities approach are worth considering as alternatives. Regardless of the approach, it is worth reporting both adjusted and raw p-values plus effect size, so that the reader can have the freedom of interpreting them.

  • 1
    $\begingroup$ The advice to present both raw and adjusted p-values has always seemed sensible to me but is it generally considered the norm, or even acceptable? $\endgroup$ – Silverfish Aug 5 '16 at 10:04

For one, it's extremely conservative. The Holm-Bonferroni method accomplishes what the Bonferonni method accomplishes (controlling the Family Wise Error Rate) while also being uniformly more powerful.

  • $\begingroup$ Is that mean that I need to use this method to correct my results or I should accept the results depending on my hypothesis. $\endgroup$ – goro Oct 16 '14 at 20:53
  • $\begingroup$ I dont know what you mean by "I should accept the results depending on my hypothesis" but yes you should apply some sort of multiple testing correction because otherwise you are highly inflating type 1 error. $\endgroup$ – TrynnaDoStat Oct 17 '14 at 12:21
  • $\begingroup$ What I meant by "I should accept the results depending on my hypothesis" is that I ran my analysis in three different ways including GLM and permutation methods. all the methods gave me significant results and those results support my hypothesis that I should have significant difference between the groups. When I used Bonferroni for multiple correction All my results was not significant. Thats why I am confused.Is this method not optimal for my analysis so I can use different one or can I trust my results depending on the results from the other methods without to use Bonferroni? $\endgroup$ – goro Oct 17 '14 at 12:32
  • 1
    $\begingroup$ Okay, I understand what you are saying. If you tested the same hypothesis 3 different ways I would not apply a multiple testing correction. The reason being that these three test results are presumably highly dependent on each other. $\endgroup$ – TrynnaDoStat Oct 17 '14 at 12:40

One should look at the "False Discovery Rate" methods as a less conservative alternative to Bonferroni. See

John D. Storey, "THE POSITIVE FALSE DISCOVERY RATE: A BAYESIAN INTERPRETATION AND THE q-VALUE," The Annals of Statistics 2003, Vol. 31, No. 6, 2013–2035.

  • 3
    $\begingroup$ These control different things though. FDR ensures that up no more 5% (or whatever your alpha is) of your calls are false positives, which is different from preserving the familywise error rate (which is what Bonferroni does). $\endgroup$ – Matt Krause Aug 5 '16 at 1:49
  • $\begingroup$ @Matt Krause: And if I understand well, the FDR (false discovery rates) do not guarantee type I error control at a predetermined level ? (see also my answer to this question) $\endgroup$ – user83346 Aug 5 '16 at 10:53

Suppose we have 20 null hypotheses, all of which happen to be true. Consider two cases:

  1. If 20 scientists each independently pick and test one of the null hypotheses at p=0.05, on average they will correctly accept 19 of the hypotheses and incorrectly reject 1.

  2. If a single scientist does one big study testing all 20 null hypotheses at p=0.05 each without the Bonferroni correction or another multiplicity correction, on average they will correctly accept 19 of the hypotheses and incorrectly reject 1.

Then the scientists in case 1 are each Bonferroni correction-compliant, but the scientist in case 2 is not. Why should the second case be considered any different from the first?

I suspect that studies testing many hypotheses are more likely to speculatively include hypotheses for which the relevant data happens to be available, e.g. "I know how strongly each person in my sample agrees with each of 20 opinions, guess I might as well check for correlation between every possible pair of opinions". So the average prior plausibility of the alternative hypotheses would be smaller. Ideally this would be handled by choosing a custom significance level to test each hypothesis at based on this prior plausibility. The Bonferroni correction could be seen as a step towards this.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.