Is the issue of multiple testing related to doing several tests on the same sample?

Question

Suppose we have a new treatment and it will be authorized only if it is effective and safe. So from a statistical point of view we have to perform two tests: one for efficacy and one for safety. We also want to have less than $\alpha = 0.05$ probability of accepting a treatment that is either ineffective or unsafe.

From what I understand from statistical test, if we run a single trial and at the end perform these two tests, we have to reduce the threshold of statistical significativity for each test (using Bonferroni correction for example) in order to keep an overall $\alpha$ at $0.05$.

But what if we run two trials, one for safety and one for efficacy? We thus have two samples, do we still need to adjust for test multiplicity?

Another question I have is that, if we forget about safety and only have to demonstrate efficacy, and two trials show different results (one test from one trial is significant and the other is not), what could we conclude about efficacy?

Answer 1

The concern over multiple testing is, at its root, a reflection of what a "significant" result actually means. A significant result means that the observed data were unlikely to have occurred due to chance if the null hypothesis is true.

If your alpha is 0.05, then roughly 1 in 20 times that you run a statistical test when the null is true you should get a significant result. So, if you run tests of 20 different treatments ⁠— none of which are effective ⁠— you would expect one of them to give you a significant result.

The question, then, is how to handle that risk. If you are testing 20 different treatments, it is obvious that you should be a little more skeptical of your results (becuase you expect one significant result by chance). The same is true for the case where you are testing two aspects of a single treatment. However, in this case you are only going to accept the treatment if both are significant. That is very different than if you were going to accept either statement (safe or efficacious).

In this case, you probably don't need to use a Bonferoni style correction. Those corrections reduce the risk of getting any significant results in a series of test. In this case, using an alpha of 0.05 already means that you have only a 0.25% chance of accepting a product that is both ineffective and unsafe (5% times 5%).

This is also an area where Bayes can be really helpful. A non-significant result does not, necessarily, mean that the treatment is dangerous (or ineffective). Having some idea about the probability that the drug is safe (or effective) can help guide how you interpret the results.

Answer 2

Technically, the probability of making at least one false positive will increase assuming the null is true. However, there is typically no need to correct for this as the two measures can be considered independent (arguments about safety and efficacy being related aside. If efficacy impacts safety, or the other way around, a more nuanced experimental design might be required).

The reason we do not correct in this instance is the same as the reason we do not correct for every other experiment we have performed in our lives; the experiments are unrelated. I think correction is applied only when tests are being performed on the same data (for example, all pairwise differences between groups).