# Family-wise error boundary: Does re-using data sets on different studies of independent questions lead to multiple testing problems?

If a team of researchers perform multiple (hypothesis) tests on a given data set, there is a volume of literature asserting that they should use some form of correction for multiple testing (Bonferroni, etc), even if the tests are independent. My question is this: does this same logic apply to multiple teams testing hypotheses on the same data set? Said another way-- what is the barrier for the family-wise error calculations? Should researchers be limited to reusing data sets for exploration only?

I disagree strongly with @fcoppens leap from recognizing the importance of multiple-hypothesis correction within a single investigation to claiming that "By the same reasoning, the same holds if several teams perform these tests."

There is no question that the more studies are performed and the more hypotheses are tested, the more Type I errors will occur. But I think there's a confusion here over the meaning of "family-wise error" rates and how they apply in actual scientific work.

First, remember that multiple-testing corrections typically arose in post-hoc comparisons for which there were no pre-formulated hypotheses. It is not at all clear that the same corrections are required when there is a small pre-defined set of hypotheses.

Second, the "scientific truth" of an individual publication does not depend on the truth of each individual statement within the publication. A well-designed study approaches an overall scientific (as opposed to statistical) hypothesis from many different perspectives, and puts together different types of results to evaluate the scientific hypothesis. Each individual result may be evaluated by a statistical test.

By the argument from @fcoppens however, if even one of those individual statistical tests makes a Type I error then that leads to a "false belief of 'scientific truth'". This is simply wrong.

The "scientific truth" of the scientific hypothesis in a publication, as opposed to the validity of an individual statistical test, generally comes from a combination of different types of evidence. Insistence on multiple types of evidence makes the validity of a scientific hypothesis robust to the individual mistakes that inevitably occur. As I look back on my 50 or so scientific publications, I would be hard pressed to find any that remains so flawless in every detail as @fcoppens seems to insist upon. Yet I am similarly hard pressed to find any where the scientific hypothesis was outright wrong. Incomplete, perhaps; made irrelevant by later developments in the field, certainly. But not "wrong" in the context of the state of scientific knowledge at the time.

Third, the argument ignores the costs of making Type II errors. A type II error might close off entire fields of promising scientific inquiry. If the recommendations of @fcoppens were to be followed, Type II error rates would escalate massively, to the detriment of the scientific enterprise.

Finally, the recommendation is impossible to follow in practice. If I analyze a set of publicly available data, I may have no way of knowing whether anyone else has used it, or for what purpose. I have no way of correcting for anyone else's hypothesis tests. And as I argue above, I shouldn't have to.

• I gave the question a bounty because I wanted to bring it 'upfront'. The reason why I wanted to do that was that I think that it doesn't get enough attention and that and that - apparently, as I experienced with my answer - there is 'no discussion' about it anymore. As shows, it may be an interesting discussion, so you get a (+1) – user83346 Aug 22 '15 at 22:31
• @fcoppens thanks for bringing this "upfront" – EdM Aug 22 '15 at 23:00
• Since this post, I stumbled upon a great paper that addresses this topic as well by Salzberg called "On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach" (cs.ru.nl/~tomh/onderwijs/lrs/lrs_files/salzberg97comparing.pdf). I appreciate the discussion. This type of questions brings up the divide between statistics and machine learning / other applied fields that was discussed in this post: stats.stackexchange.com/questions/1194/… .... – toypajme Aug 24 '15 at 0:00
• A paper by Breiman also addresses this topic: projecteuclid.org/euclid.ss/1009213726 . I hope these papers can serve as an easy reference for those who are interested in the current research and published discussions on this topic. – toypajme Aug 24 '15 at 0:01
• There is also the following paper "On the generation and ownership of alpha in medical studies". It is clearly a controversial topic. One of the few cases where the answer is clear is with regulatory label claims for pharmaceutical products, where there is more or less a single $\alpha=0.05$ for a medical study. Once we get into scientific publications, there is for better or wose nobody to enforce any such thing. – Björn Aug 27 '15 at 10:56

The ''multiple testing'' correction is necessary whenever you 'inflate the type I error': e.g. if you perform two tests, each at a confidence level $\alpha=5\%$, and for the first we test the null $H_0^{(1)}$ against the alternative $H_1^{(1)}$ and the second hypothesis $H_0^{(2)}$ versus $H_1^{(2)}$.

Then we know that the type I error for e.g. the first hypothesis is the probability of falsely rejecting $H_0^{(1)}$ and is it $\alpha=5\%$.

If you perform the two tests, then the probability that at least one of the two is falsely rejected is equal to the 1 minus the probability that both are accepted so $1 - (1-\alpha)^2$ which, for $\alpha=5\%$ is equal to $9.75\%$, so the type one error of having at least one false rejection is has almost doubled !

In statistical hypothesis testing one can only find statistical evidence for the alternative hypothesis by rejecting the null, rejecting the null allows us to conclude that there is evidence in favour of the alternative hypothesis. (see also What follows if we fail to reject the null hypothesis?).

So a false rejection of the null gives us false evidence so a false belief of ''scientific truth''. This is why this type I inflation (the almost doubling of the type I error) has to be avoided; higher type I errors imply more false beliefs that something is scientifically proven. Therefore people ''control'' the type Ierror at a familywise level.

If there is a team of researchers that performs multiple tests, then each time they reject the null hypothesis they conclude that they have found statitiscal evidence of a scientific truth. However, by the above, many more than $5\%$ of these conclusions are a false believe of ''scientific truth''.

By the same reasoning, the same holds if several teams perform these tests (on the same data).

Obviously, the above findings only hold if we the teams work on the same data. What is different then when they work on different samples ?

To explain this, let's take a simple and very unrealistic example. Our null hypothesis is that a population has a normal distribution, with known $\sigma$ and the null states that $H_0: \mu = 0$ against $H_1: \mu \ne 0$. Let's take the significance level $\alpha=5\%$.

Our sample ('the data') is only one observation, so we will reject the null when the observation $o$ is either larger than $1.96\sigma$ or smaller than $-1.96\sigma$.

We make a type I error with a probability of $5\%$ because it could be that we reject $H_0$ just by chance, indeed, if $H_0$ is true (so the population is normal and $\mu=0$) then there is (with $H_0$ true) a chance that $o \not \in [-1.96\sigma;1.96\sigma$]. So even if $H_0$ is true then there is a chance that we have bad luck with the data.

So if we use the same data, it could be that the conclusions of the tests are based on a sample that was drawn with ''bad chance''. With another sample the context is different.

• I am not a fan of using "proof" with respect to scientific evidence. – Alexis Aug 17 '15 at 16:44
• @Alexis: it is certainly because English is not my native language, but I thought that 'evidence' and 'proof' are more or like synomym, but that does not seem to be the case ? – user83346 Aug 17 '15 at 19:55
• Formal "proof," in my opinion, belongs in mathematics. Or, less formally, belongs in jurisprudence. To me proof does not belong to science, because that implies the end of inquiry and the start of dogma, and science is fundamentally about inquiry. In English (and in the USA), for example, we have a rhetorical game where anti-evolution individuals will say "biological evolution is just a theory, and hasn't been scientifically proven." Of course, the trick is getting the listeners to forget that science never proves, only provides evidence. – Alexis Aug 18 '15 at 1:57