I am wondering why it is said that multiple testing corrections are ''arbitrary'' and that they are based on a incoherent philosophy that

the veracity of one statement depends on which other hypotheses are entertained

see e.g. answers and comments to What's wrong with Bonferroni adjustments? and in particular the discussion between @FrankHarrell and @Bonferroni.

Let us (for simplicity and for the ease of the exposition) assume that we have two (independent) normal populations, independent and with known standard deviations but unknown means. Let (just as an example) say that these standard deviations are resp. $\sigma_1=2, \sigma_2=3$.

Joint test

Assume we want to test the hypothesis $H_0: \mu_1 = 2 \& \mu_2=2$ versus $H_1: \mu_1 \ne 2 | \mu_2 \ne 2$ at a significance level of $\alpha=0.05$ (the symbol $\&$ meaning 'and' while $|$ means 'or').

We also have a random outcome $x_1$ from the first population and $x_2$ from the second population.

if $H_0$ is true then the first random variable $X_1 \sim N(\mu_1=2,\sigma_1=2)$ and the second one $X_2 \sim N(\mu_2=2,\sigma_2=3)$ as we assumed independence it holds that the random variable $X^2 = \frac{(X_1-\mu_1)^2}{\sigma_1^2} + \frac{(X_2-\mu_2)^2}{\sigma_2^2}$ is $\chi^2$ with $df=2$. We can use this $X^2$ as a test statistic and we will accept $H_0$ if, for the observed outcomes $x_1$ and $x_2$ it holds that $\frac{(x_1-\mu_1)^2}{\sigma_1^2} + \frac{(x_2-\mu_2)^2}{\sigma_2^2} \le \chi^2_\alpha$. In other words the acceptance region for this test is an ellipse centered at $(\mu_1, \mu_2)$ and we have a density mass of $1-\alpha$ ''on top'' of this ellipse.

Multiple tests

With multiple testing we will do two independent tests and ''adjust'' the significance level. So we will perform two independent tests $H_0^{(1)}: \mu_1 = 2$ versus $H_1^{(1)}: \mu_1 \ne 2$ and a second test $H_0^{(2)}: \mu_2 = 2$ versus $H_1^{(2)}: \mu_2 \ne 2$ but with an adjusted significance level $\alpha^{adj.}$ that is such that $1-(1-\alpha^{adj.})^2=0.05$ or $(1-\alpha^{adj.})^2=0.95$ or $1-\alpha^{adj.}=\sqrt{0.95}$ or $\alpha^{adj.}=1-\sqrt{0.95}$ which yields $\alpha^{adj.}=0.02532057$.

In this case we will accept $H_0^{(1)}$ and $H_0^{(1)}$ (and both together are equivalent to our ''original'' $H_0: \mu_1 = 2 \& \mu_2=2$) whenever $\frac{x_1 - \mu_1}{\sigma_1} \le z_{\alpha^{adj.}} $ and $\frac{x_2 - \mu_2}{\sigma_2} \le z_{\alpha^{adj.}} $

So we conclude that, with multiple testing, the acceptance region for $x_1,x_2$ has become a rectangle with center $(\mu_1,\mu_2)$ and with a probability mass of $1-\alpha$ on top of it.


So we find that, for a joint ($\chi^2$) test the geometrical shape of the acceptance region is an ellipse, while with multiple testing it is a rectangle. The density mass ''on top'' of the acceptance region is in both cases 0.95.


So what is then the problem with multiple testing ? If there exists such a problem, then (see supra) the same problem should exist for joint tests or not ? The reason can not be that we prefer ellipses over rectangles does it ?


3 Answers 3


I think you are missing @FrankHarrell's point here (I do not currently have access to the Perneger's paper discussed in the linked thread, so cannot comment on it).

The debate is not about math, it is about philosophy. Everything you wrote here is mathematically correct, and clearly Bonferroni correction allows to control the familywise type I error rate, as your "joint test" also does. The debate is not at all about the specifics of Bonferroni itself, it is about multiple testing adjustments in general.

Everybody knows an argument for multiple testing corrections, as illustrated by the famous XKCD jelly beans comic:

enter image description here

Here is a counter-argument: if I developed a really convincing theory predicting that specifically green jelly beans should cause acne; and if I ran experiment to test for it and got nice and clear $p=0.003$; and if it so happened that some other PhD student in the same lab for whatever reason ran nineteen tests for all other jelly beans colors getting $p>05$ every time; and if now our advisor wants to put all of that in one single paper; -- then I would be totally against "adjusting" my p-value from $p=0.003$ to $p=0.003\cdot 20 = 0.06$.

Note that the experimental data in the Argument and in the Counter-Argument might be exactly the same. But the interpretation differs. This is fine, but illustrates that one should not be obliged by doing multiple testing corrections in all situations. It is ultimately a matter of judgment. Crucially, real-life scenarios are usually not as clear cut as here and tend to be in between #1 and #2. See also Frank's example in his answer.

  • $\begingroup$ I must say that I don't fully get the poitnt here and I would really like to understand so I try to make my point; I think that it all depends on your hypothesis that you're testing, I don't see the exact hypothesis that's being tested for the 'other PhD student' in your example and in my opinion it all depends on that. I tried to explain my point in the 'edit' section at the bottom of this answer, where I take mr Harrell's example and try to make my point: stats.stackexchange.com/questions/120362/… $\endgroup$
    – user83346
    Commented Aug 11, 2016 at 14:44
  • 1
    $\begingroup$ Sure it depends on the hypothesis, @fcop. The problem is that in real life the hypotheses are not always as clear as in maths. Somebody can have a "hypothesis" that treatment A will outperform treatment B. But then various measures of performance are collected, and the usual garden of forking paths begins. If I stupidly recorded 10 measures and am trying all of them, then it's clearly scenario #1 in my answer. If I have a clear a priori choice then it's #2. But usually I have some hunch preferences but am not really sure and then my advisor suggests to try that other measure too etc. etc... $\endgroup$
    – amoeba
    Commented Aug 11, 2016 at 14:58
  • $\begingroup$ yes but I think that ''in real life'' there is a perfectly sound explanantion for the example with the yellow beans, only it is to long to put in a comment, may I put it in an answer ? $\endgroup$
    – user83346
    Commented Aug 11, 2016 at 15:13
  • $\begingroup$ I posted an answer to make my point, please feel free to react critically because I really want to understand. $\endgroup$
    – user83346
    Commented Aug 11, 2016 at 16:03
  • 1
    $\begingroup$ @fcop I agree with amoeba here. For me the thing about your question is that is exclusively couched in abstract terms. If on the "Is statistics part of mathematics or part of science?" continuum you come from the =science end as I do then your question leaves out everything which is important. $\endgroup$
    – mdewey
    Commented Aug 13, 2016 at 12:53

@amoeba: on the example with the jelly beans I would like to argue as follows (note, I just want to understand):

Let's say that there are 20 different colors of jelly beans, let's call these $c_1, c_2, \dots , c_{20}$, and let $c_{10}$ be the color 'green'.

So, with your example the p-values for color $i$ (we note this as $p^{(i)}$) will be $p^{(i)} > 0.05$ when $i \ne 10$ and $p^{(10)}=0.003$.

  1. Theory 1: green jelly beans cause acne

    If you have developed a theory that green jelly beans cause acne, then you should test the hypothesis

    $H_0$: ''jelly beans of color $c_{10}$ have no effect on acne'' versus $H_1$: ''jelly beans of color $c_{10}$ cause acne''. This is obviously not a multiple testing problem, so you do not have to adjust the p-values.

  2. Theory 2: only green jelly beans cause acne

    In that case you should have ''$H_1$: green jelly beans cause acne AND jelly beans of color $c_i, i\ne 10$ do not cause acne'' and $H_0$ is then ''green jelly beans do not cause acne OR $\exists i|i \ne 10$ such that beans of color $c_i$ cause acne''.

    This is a multiple testing problem and requires adjusted p-values.

  3. Theory 3: jelly beans (of whatever color) cause acne

    In that case $H_1$: ''jelly beans of color $c_1$ cause acne AND ''jelly beans of color $c_2$ cause acne AND .... AND ''jelly beans of color $c_{20}$ cause acne'' and $H_0$ is the opposite.

    This is again a multiple testing problem.

  4. Theory ...


Anyhow, it can be seen that these theories are fundamentally different and whether or not p-value adjustment is required depends on that, not on ''philosophy'', at least that is my understanding.

P.S. for the reaction to the example of @FrankHarrell see ''EDIT'' at the bottom of my answer to What's wrong with Bonferroni adjustments?

  • 1
    $\begingroup$ Everything you write is correct but only applicable in the ideal textbook situation when $H_0$ is precisely formulated in advance. In practice this is simply not the case most of the times. The jelly beans example is a bit silly, but here it goes: the theory is that greenish beans cause acne. Note that it is vague. You try green jelly beans. Your boss tells you to try olive ones and emerald green ones too. You comply. Now, you find p=0.02, p=0.3 and p=0.3. What do you do? Correct for three tests? Or not? What would you have done if it were 0.3, 0.02, and 0.3 instead? $\endgroup$
    – amoeba
    Commented Aug 13, 2016 at 11:11
  • $\begingroup$ Your boss says: we ran three experiments, we need to use Bonferroni, nothing is significant. You say: but I did not even want to test olive and emerald, so no need for any corrections, I have a significant result. Note that research hypothesis was not precise enough ("greenish") to tell us if it was talking about "green" or "green+olive+emerald". Everything hinges on what would you have done if only emerald came out significant. Would you simply ignore it (not part of your hypothesis), or use Bonferroni (to try to salvage it)? Hard to say in advance! $\endgroup$
    – amoeba
    Commented Aug 13, 2016 at 11:15
  • $\begingroup$ Again, this jelly beans example gets a bit silly by now (this example is probably more sensible; I might edit my answer to elaborate on it), but that's how most of the research is being done. Most of it is kinda between confirmatory and exploratory. $\endgroup$
    – amoeba
    Commented Aug 13, 2016 at 11:17
  • 1
    $\begingroup$ @amoeba: but we seem to agree that it is no longer about ''philiosophy'' ? If my boss said that I should try green olive and emarald, then, me as a statistician should ask my boss: but what do you want to ''show'' that green and olive and emerald beens cause acne or that one of them causes acne ? P.S. The link 'this example' does not seem to work ? For exploratory research I think you should use FDR in stead of hypothesis testing ? $\endgroup$
    – user83346
    Commented Aug 13, 2016 at 11:31
  • 1
    $\begingroup$ @amoeba: I think you can do data mining in your search for 'new theories' but once you discover a new theory you should find confirmation for it using ANOTHER sample than the sample you used for data mining. What do you think? $\endgroup$
    – user83346
    Commented Aug 15, 2016 at 15:02

I'll leave my old answer at the end to provide context for your comment.

It seems to me that your rectangular-versus-ellipsoid thought experiment gives an interesting hint of a problem with multiple comparisons: your multiple test example is in some sense projecting information down in dimensionality, then back up, losing information in the process.

That is, the joint probability is ellipsoid precisely because you have two Gaussian distributions, which will jointly yield an ellipsoid, whose circularity is determined by the relative variance of the two distributions, and whose major axis' slope is determined by the correlation of the two sets of data. Since you specify the two datasets are independent, the major axis is parallel to the x or y axis.

On the other hand, your two-test example projects Gaussian distributions down to a 1-D range and when you then combine the two tests into a single, 2-D graph (projecting back up), you have lost information and the resulting 95% area is a rectangular rather than the appropriate ellipsoid. And things get worse if the two datasets are correlated.

So it seems to me that this might be an indication that multiple testing is losing information due to what we might describe as projecting information down -- losing information in the process -- then back up. So the shape of the resulting pseudo-joint density is incorrect and attempting to scale its axes via something like a Boneferroni can't fix that.

So in answer to your question, I'd say yes, we prefer an ellipse in our joint distribution rather than the incorrect (due to loss of information) rectangle of our pseudo-joint distribution. Or perhaps the issue is that you've created a pseudo-joint density in the first place.

BUT your question is more philosophical than that, and I have to support Amoeba's answer that it's not simply a matter of the math. For example, what if you pre-registered your jellybean experiment with a precise "green jelly beans" as part of your hypothesis, rather than an imprecise "greenish". You perform the experiment and find no statistically significant effect. Then your lab assistant shows you a photo they took of themselves in front of all of the jellybean doses -- what a Herculean task they performed! And something you say leads the assistant to realize that you are partially colorblind.

It turns out that what you called "green" we're actually green and aqua jellybeans! With the help of the photo, the assistant properly codes the results and it turns out green jellybeans are significant! Your career is saved! Except you've just done a multiple comparison: you took two swipes at the data, and if you had found significance in the first place, no one would have ever known any different.

This isn't a matter of you p-value-hacking. It was an honest correction, but your motivation doesn't matter here.

And if we're being totally honest, "green" is no more specific than "greenish". First, in terms of the actual color, and then in terms of the fact that green is most likely a proxy for other ingredients.

And what if you had never discovered your error, but for some reason your assistant replicated the experiment and the second results were significant? Basically the same case, though you did collect two sets of data. At this point, I'm starting to wander, so let me summarize by again saying I believe Amoeba has it right and your "it is or isn't because of mathematics" idea is technically correct, but not tractable in the real world.

OLD answer: Is this question actually about correlation? I'm thinking more of a Mahalanobis Distance kind of issue, where independently looking at the 95% x1 and the 95% x2 yields a rectangle, but this assumes that x1 and x2 are not correlated. While using the Mahalanobis Distance (an ellipse that is shaped based on the correlation between x1 and x2) is superior. The ellipse extends outside of the rectangle, so it accepts some points that are outside of the rectangle, but it also rejects points inside the rectangle. Assuming x1 and x2 are correlated to some degree.

Otherwise, if you assume x1 and x2 have 0 correlation, what distribution are you assuming for each? If uniform, you'd get a rectangular region, if normal you'll get an elliptical region. Again, this would be independent of multiple testing corrections or not.

  • 1
    $\begingroup$ Thanks for your try, but this is about multiple testing (like e.g. Bonferroni and other). As I stated in the answer, we assume independence so it's not about correlation in this question. Dependence would be an interesting extension of the problem but I will first try to get some answers assuming independence. $\endgroup$
    – user83346
    Commented Aug 10, 2016 at 17:10
  • $\begingroup$ I do not get the point, I never said something about greenish ? And I agree, if we start with ''what if you are color blind'' then I agree that it becomes philosofical, by the way, if the assistant finds in some way that I am color blind, couldn't it be that the color blindness is with the assistant and not with me ? $\endgroup$
    – user83346
    Commented Aug 13, 2016 at 15:07
  • $\begingroup$ @fcop: Re: "greenish", you may not mention it but Amoeba does in his interactions with you in his comments. I am agreeing with him and saying it's even worse than the case he makes. Who is colorblind doesn't matter, I was simply trying to make a more realistic scenario where a multiple comparison might masquerade as something else. $\endgroup$
    – Wayne
    Commented Aug 13, 2016 at 17:53
  • $\begingroup$ an assistant discovering ,after a hypothesis test, that the researcher is colorblind is a ''more realistic scenario'' ? $\endgroup$
    – user83346
    Commented Aug 14, 2016 at 6:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.