Family-wise error boundary: Does re-using data sets on different studies of independent questions lead to multiple testing problems? If a team of researchers perform multiple (hypothesis) tests on a given data set, there is a volume of literature asserting that they should use some form of correction for multiple testing (Bonferroni, etc), even if the tests are independent. My question is this: does this same logic apply to multiple teams testing hypotheses on the same data set? Said another way-- what is the barrier for the family-wise error calculations? Should researchers be limited to reusing data sets for exploration only?
 A: The ''multiple testing'' correction is necessary whenever you 'inflate the type I error':  e.g. if you perform two tests, each at a confidence level $\alpha=5\%$, and for the first we test the null $H_0^{(1)}$ against the alternative $H_1^{(1)}$ and the second hypothesis $H_0^{(2)}$ versus $H_1^{(2)}$.  
Then we know that the type I error for e.g. the first hypothesis is the probability of falsely rejecting $H_0^{(1)}$ and is it $\alpha=5\%$. 
If you perform the two tests, then the probability that at least one of the two is falsely rejected is equal to the 1 minus the probability that both are accepted so $1 - (1-\alpha)^2$ which, for $\alpha=5\%$ is equal to $9.75\%$, so the type one error of having at least one false rejection is has almost doubled !
In statistical hypothesis testing one can only find statistical evidence for the alternative hypothesis by rejecting the null, rejecting the null allows us to conclude that there is evidence in favour of the alternative hypothesis. (see also What follows if we fail to reject the null hypothesis?).  
So a false rejection of the null gives us false evidence so a false belief of ''scientific truth''.  This is why this type I inflation (the almost doubling of the type I error) has to be avoided; higher type I errors imply more false beliefs that something is scientifically proven.  Therefore people ''control'' the type Ierror at a familywise level. 
If there is a team of researchers that performs multiple tests, then each time they reject the null hypothesis they conclude that they have found statitiscal evidence of a scientific truth.  However, by the above, many more than $5\%$ of these conclusions are a false believe of ''scientific truth''.  
By the same reasoning, the same holds if several teams perform these tests (on the same data). 
Obviously, the above findings only hold if we the teams work on the same data.  What is different then when they work on different samples ?
To explain this, let's take a simple and very unrealistic example.  Our null hypothesis is that a population has a normal distribution, with known $\sigma$ and the null states that $H_0: \mu = 0$ against $H_1: \mu \ne 0$. Let's take the significance level $\alpha=5\%$. 
Our sample ('the data') is only one observation, so we will reject the null when the observation $o$ is either larger than $1.96\sigma$ or smaller than $-1.96\sigma$. 
We make a type I error with a probability of $5\%$ because it could be that we reject $H_0$ just by chance, indeed, if $H_0$ is true (so the population is normal and $\mu=0$) then there is (with $H_0$ true) a chance that $o \not \in [-1.96\sigma;1.96\sigma$]. 
So even if $H_0$ is true then there is a chance that we have bad luck with the data.  
So if we use the same data, it could be that the conclusions of the tests are based on a sample that was drawn with ''bad chance''.  With another sample the context is different. 
A: I disagree strongly with @fcoppens leap from recognizing the importance of multiple-hypothesis correction within a single investigation to claiming that "By the same reasoning, the same holds if several teams perform these tests."
There is no question that the more studies are performed and the more hypotheses are tested, the more Type I errors will occur. But I think there's a confusion here over the meaning of "family-wise error" rates and how they apply in actual scientific work.
First, remember that multiple-testing corrections typically arose in post-hoc comparisons for which there were no pre-formulated hypotheses. It is not at all clear that the same corrections are required when there is a small pre-defined set of hypotheses.
Second, the "scientific truth" of an individual publication does not depend on the truth of each individual statement within the publication. A well-designed study approaches an overall scientific (as opposed to statistical) hypothesis from many different perspectives, and puts together different types of results to evaluate the scientific hypothesis. Each individual result may be evaluated by a statistical test.
By the argument from @fcoppens however, if even one of those individual statistical tests makes a Type I error then that leads to a "false belief of 'scientific truth'". This is simply wrong.
The "scientific truth" of the scientific hypothesis in a publication, as opposed to the validity of an individual statistical test, generally comes from a combination of different types of evidence. Insistence on multiple types of evidence makes the validity of a scientific hypothesis robust to the individual mistakes that inevitably occur. As I look back on my 50 or so scientific publications, I would be hard pressed to find any that remains so flawless in every detail as @fcoppens seems to insist upon. Yet I am similarly hard pressed to find any where the scientific hypothesis was outright wrong. Incomplete, perhaps; made irrelevant by later developments in the field, certainly. But not "wrong" in the context of the state of scientific knowledge at the time.
Third, the argument ignores the costs of making Type II errors. A type II error might close off entire fields of promising scientific inquiry. If the recommendations of @fcoppens were to be followed, Type II error rates would escalate massively, to the detriment of the scientific enterprise.
Finally, the recommendation is impossible to follow in practice. If I analyze a set of publicly available data, I may have no way of knowing whether anyone else has used it, or for what purpose. I have no way of correcting for anyone else's hypothesis tests. And as I argue above, I shouldn't have to.
