What might be a clear, practical definition for a “family of hypotheses” (with respect to familywise error rate)?

When trying to evaluate what constitutes a family of hypotheses within an experiment/project/analysis, I've found "similar in purpose" and "similar in content" given as guidelines for delimiting families, but these leave quite a lot open to interpretation (to say the least).

It seems clear that if in the course of an analysis, I do several tests of group means and a separate batch of tests of homogeneity of proportions, that I wouldn't bundle everything up together into a single family of hypotheses.

However, if I have several batches of somewhat related tests of group means, what criterion brings them together into a family (or splits them into separate families)? Should all members of a family have the same response variable? If I had different response variables but the same set of cases involved, would those all get bundled into a family of hypotheses?

The issue of multiple comparisons is a really big topic. There have been many opinions and many disagreements. This is due to many things; among others, it is partly because the issue is really important, and partly because there really is no ultimate rule or criterion. Take a prototypical case: You conduct an experiment with $k$ treatments and get a significant ANOVA, so now you wonder which treatment means differ. How should you go about this, run $k(k-1)/2$ t-tests? Although these tests would individually hold $\alpha$ at .05, the 'familywise' $\alpha$ (i.e., the probability that at least 1 type I error will occur) will explode. In fact, the familywise error rate will be $1-(1-\alpha)^k$. The question is, what defines a 'family'? And there is no ultimate answer, beyond the trivial one that a 'family' is a set of contrasts. Whether any particular set of contrasts should be considered a family is a subjective decision. The 3rd, 17th, and 42nd analyses that I ever conducted in my life are a set of contrasts, and I could have adjusted my $\alpha$ threshold to insure that the probability of type I errors amongst them was held at 5%, but no one would find this sensical. The question for you is whether you consider your contrasts to be a set in a meaningful sense, and only you can make that judgment. I will offer some standard approaches. Many analysts believe that if a set of contrasts come from the same experiment / data set, they should be treated as a family, and procedures (such as $\alpha$ adjustment) are necessary. Others believe that even when contrasts come from the same experiment, if they are a-priori and orthogonal, special procedures are not required. Both of these positions can be defended. Finally, note also that procedures to control familywise error rates come at a cost--viz. increased type II error rates.

The criterion is that the hypotheses are interdependent in the sense that if one of them breaks then the whole your conclusion or theory breaks. Hence you need a guarantee that if all the tests are significant none of them is significant falsely.

• So, running thousands of t-tests across different measurements in a before-and-after-treatment experiment (like a gene expression experiment) would not count as a family of tests? One false positive wouldn't be desirable, but it wouldn't completely break the conclusions from the experiment as a whole. – Ryan Dec 19 '11 at 18:37
• I think so. If that wasn't sound a statistician should wish to die young or quit profession soon, in order to escape multiplying type I error in his life course. – ttnphns Dec 19 '11 at 19:36
• Ok, alright. Following strict boolean logic in a world where all problems are like those encountered in casino and other simple games, one type I error certainly would invalidate the entire theory. – Ryan Dec 20 '11 at 2:18

A discussion on researchgate (http://www.researchgate.net/post/Bonferroni-how_is_the_family_of_hypotheses_defined) provided a list of papers, which might help collecting opinions - the papers actually start from the question "when to apply corrections in a multiple testing situation". The papers -all cited often - are:

1) Rothman KJ. No adjustments are needed for multiple comparisons. Epidemiology.1990;1(1):43-6. http://psg-mac43.ucsf.edu/ticr/syllabus/courses/9/2003/02/27/Lecture/readings/Rothman.pdf