What's wrong with ''multiple testing correction'' compared to ''joint tests''?

Question

I am wondering why it is said that multiple testing corrections are ''arbitrary'' and that they are based on a incoherent philosophy that

the veracity of one statement depends on which other hypotheses are entertained

see e.g. answers and comments to What's wrong with Bonferroni adjustments? and in particular the discussion between @FrankHarrell and @Bonferroni.

Let us (for simplicity and for the ease of the exposition) assume that we have two (independent) normal populations, independent and with known standard deviations but unknown means. Let (just as an example) say that these standard deviations are resp. $\sigma_1=2, \sigma_2=3$.

Joint test

Assume we want to test the hypothesis $H_0: \mu_1 = 2 \& \mu_2=2$ versus $H_1: \mu_1 \ne 2 | \mu_2 \ne 2$ at a significance level of $\alpha=0.05$ (the symbol $\&$ meaning 'and' while $|$ means 'or').

We also have a random outcome $x_1$ from the first population and $x_2$ from the second population.

if $H_0$ is true then the first random variable $X_1 \sim N(\mu_1=2,\sigma_1=2)$ and the second one $X_2 \sim N(\mu_2=2,\sigma_2=3)$ as we assumed independence it holds that the random variable $X^2 = \frac{(X_1-\mu_1)^2}{\sigma_1^2} + \frac{(X_2-\mu_2)^2}{\sigma_2^2}$ is $\chi^2$ with $df=2$. We can use this $X^2$ as a test statistic and we will accept $H_0$ if, for the observed outcomes $x_1$ and $x_2$ it holds that $\frac{(x_1-\mu_1)^2}{\sigma_1^2} + \frac{(x_2-\mu_2)^2}{\sigma_2^2} \le \chi^2_\alpha$. In other words the acceptance region for this test is an ellipse centered at $(\mu_1, \mu_2)$ and we have a density mass of $1-\alpha$ ''on top'' of this ellipse.

Multiple tests

With multiple testing we will do two independent tests and ''adjust'' the significance level. So we will perform two independent tests $H_0^{(1)}: \mu_1 = 2$ versus $H_1^{(1)}: \mu_1 \ne 2$ and a second test $H_0^{(2)}: \mu_2 = 2$ versus $H_1^{(2)}: \mu_2 \ne 2$ but with an adjusted significance level $\alpha^{adj.}$ that is such that $1-(1-\alpha^{adj.})^2=0.05$ or $(1-\alpha^{adj.})^2=0.95$ or $1-\alpha^{adj.}=\sqrt{0.95}$ or $\alpha^{adj.}=1-\sqrt{0.95}$ which yields $\alpha^{adj.}=0.02532057$.

In this case we will accept $H_0^{(1)}$ and $H_0^{(1)}$ (and both together are equivalent to our ''original'' $H_0: \mu_1 = 2 \& \mu_2=2$) whenever $\frac{x_1 - \mu_1}{\sigma_1} \le z_{\alpha^{adj.}} $ and $\frac{x_2 - \mu_2}{\sigma_2} \le z_{\alpha^{adj.}} $

So we conclude that, with multiple testing, the acceptance region for $x_1,x_2$ has become a rectangle with center $(\mu_1,\mu_2)$ and with a probability mass of $1-\alpha$ on top of it.

Conclusion

So we find that, for a joint ($\chi^2$) test the geometrical shape of the acceptance region is an ellipse, while with multiple testing it is a rectangle. The density mass ''on top'' of the acceptance region is in both cases 0.95.

Questions

So what is then the problem with multiple testing ? If there exists such a problem, then (see supra) the same problem should exist for joint tests or not ? The reason can not be that we prefer ellipses over rectangles does it ?

Answer 1

I think you are missing @FrankHarrell's point here (I do not currently have access to the Perneger's paper discussed in the linked thread, so cannot comment on it).

The debate is not about math, it is about philosophy. Everything you wrote here is mathematically correct, and clearly Bonferroni correction allows to control the familywise type I error rate, as your "joint test" also does. The debate is not at all about the specifics of Bonferroni itself, it is about multiple testing adjustments in general.

Everybody knows an argument for multiple testing corrections, as illustrated by the famous XKCD jelly beans comic:

Here is a counter-argument: if I developed a really convincing theory predicting that specifically green jelly beans should cause acne; and if I ran experiment to test for it and got nice and clear $p=0.003$; and if it so happened that some other PhD student in the same lab for whatever reason ran nineteen tests for all other jelly beans colors getting $p>05$ every time; and if now our advisor wants to put all of that in one single paper; -- then I would be totally against "adjusting" my p-value from $p=0.003$ to $p=0.003\cdot 20 = 0.06$.

Note that the experimental data in the Argument and in the Counter-Argument might be exactly the same. But the interpretation differs. This is fine, but illustrates that one should not be obliged by doing multiple testing corrections in all situations. It is ultimately a matter of judgment. Crucially, real-life scenarios are usually not as clear cut as here and tend to be in between #1 and #2. See also Frank's example in his answer.

Answer 2

@amoeba: on the example with the jelly beans I would like to argue as follows (note, I just want to understand):

Let's say that there are 20 different colors of jelly beans, let's call these $c_1, c_2, \dots , c_{20}$, and let $c_{10}$ be the color 'green'.

So, with your example the p-values for color $i$ (we note this as $p^{(i)}$) will be $p^{(i)} > 0.05$ when $i \ne 10$ and $p^{(10)}=0.003$.

1. Theory 1: green jelly beans cause acne

   If you have developed a theory that green jelly beans cause acne, then you should test the hypothesis

   $H_0$: ''jelly beans of color $c_{10}$ have no effect on acne'' versus $H_1$: ''jelly beans of color $c_{10}$ cause acne''. This is obviously not a multiple testing problem, so you do not have to adjust the p-values.

2. Theory 2: only green jelly beans cause acne

   In that case you should have ''$H_1$: green jelly beans cause acne AND jelly beans of color $c_i, i\ne 10$ do not cause acne'' and $H_0$ is then ''green jelly beans do not cause acne OR $\exists i|i \ne 10$ such that beans of color $c_i$ cause acne''.

   This is a multiple testing problem and requires adjusted p-values.

3. Theory 3: jelly beans (of whatever color) cause acne

   In that case $H_1$: ''jelly beans of color $c_1$ cause acne AND ''jelly beans of color $c_2$ cause acne AND .... AND ''jelly beans of color $c_{20}$ cause acne'' and $H_0$ is the opposite.

   This is again a multiple testing problem.

4. Theory ...

Conclusion

Anyhow, it can be seen that these theories are fundamentally different and whether or not p-value adjustment is required depends on that, not on ''philosophy'', at least that is my understanding.

P.S. for the reaction to the example of @FrankHarrell see ''EDIT'' at the bottom of my answer to What's wrong with Bonferroni adjustments?

Answer 3

I'll leave my old answer at the end to provide context for your comment.

It seems to me that your rectangular-versus-ellipsoid thought experiment gives an interesting hint of a problem with multiple comparisons: your multiple test example is in some sense projecting information down in dimensionality, then back up, losing information in the process.

That is, the joint probability is ellipsoid precisely because you have two Gaussian distributions, which will jointly yield an ellipsoid, whose circularity is determined by the relative variance of the two distributions, and whose major axis' slope is determined by the correlation of the two sets of data. Since you specify the two datasets are independent, the major axis is parallel to the x or y axis.

On the other hand, your two-test example projects Gaussian distributions down to a 1-D range and when you then combine the two tests into a single, 2-D graph (projecting back up), you have lost information and the resulting 95% area is a rectangular rather than the appropriate ellipsoid. And things get worse if the two datasets are correlated.

So it seems to me that this might be an indication that multiple testing is losing information due to what we might describe as projecting information down -- losing information in the process -- then back up. So the shape of the resulting pseudo-joint density is incorrect and attempting to scale its axes via something like a Boneferroni can't fix that.

So in answer to your question, I'd say yes, we prefer an ellipse in our joint distribution rather than the incorrect (due to loss of information) rectangle of our pseudo-joint distribution. Or perhaps the issue is that you've created a pseudo-joint density in the first place.

BUT your question is more philosophical than that, and I have to support Amoeba's answer that it's not simply a matter of the math. For example, what if you pre-registered your jellybean experiment with a precise "green jelly beans" as part of your hypothesis, rather than an imprecise "greenish". You perform the experiment and find no statistically significant effect. Then your lab assistant shows you a photo they took of themselves in front of all of the jellybean doses -- what a Herculean task they performed! And something you say leads the assistant to realize that you are partially colorblind.

It turns out that what you called "green" we're actually green and aqua jellybeans! With the help of the photo, the assistant properly codes the results and it turns out green jellybeans are significant! Your career is saved! Except you've just done a multiple comparison: you took two swipes at the data, and if you had found significance in the first place, no one would have ever known any different.

This isn't a matter of you p-value-hacking. It was an honest correction, but your motivation doesn't matter here.

And if we're being totally honest, "green" is no more specific than "greenish". First, in terms of the actual color, and then in terms of the fact that green is most likely a proxy for other ingredients.

And what if you had never discovered your error, but for some reason your assistant replicated the experiment and the second results were significant? Basically the same case, though you did collect two sets of data. At this point, I'm starting to wander, so let me summarize by again saying I believe Amoeba has it right and your "it is or isn't because of mathematics" idea is technically correct, but not tractable in the real world.

OLD answer: Is this question actually about correlation? I'm thinking more of a Mahalanobis Distance kind of issue, where independently looking at the 95% x1 and the 95% x2 yields a rectangle, but this assumes that x1 and x2 are not correlated. While using the Mahalanobis Distance (an ellipse that is shaped based on the correlation between x1 and x2) is superior. The ellipse extends outside of the rectangle, so it accepts some points that are outside of the rectangle, but it also rejects points inside the rectangle. Assuming x1 and x2 are correlated to some degree.

Otherwise, if you assume x1 and x2 have 0 correlation, what distribution are you assuming for each? If uniform, you'd get a rectangular region, if normal you'll get an elliptical region. Again, this would be independent of multiple testing corrections or not.