What might be a clear, practical definition for a “family of hypotheses” (with respect to familywise error rate)?

Question

When trying to evaluate what constitutes a family of hypotheses within an experiment/project/analysis, I've found "similar in purpose" and "similar in content" given as guidelines for delimiting families, but these leave quite a lot open to interpretation (to say the least).

It seems clear that if in the course of an analysis, I do several tests of group means and a separate batch of tests of homogeneity of proportions, that I wouldn't bundle everything up together into a single family of hypotheses.

However, if I have several batches of somewhat related tests of group means, what criterion brings them together into a family (or splits them into separate families)? Should all members of a family have the same response variable? If I had different response variables but the same set of cases involved, would those all get bundled into a family of hypotheses?

Answer 1

The issue of multiple comparisons is a really big topic. There have been many opinions and many disagreements. This is due to many things; among others, it is partly because the issue is really important, and partly because there really is no ultimate rule or criterion. Take a prototypical case: You conduct an experiment with $k$ treatments and get a significant ANOVA, so now you wonder which treatment means differ. How should you go about this, run $k(k-1)/2$ t-tests? Although these tests would individually hold $\alpha$ at .05, the 'familywise' $\alpha$ (i.e., the probability that at least 1 type I error will occur) will explode. In fact, the familywise error rate will be $1-(1-\alpha)^k$. The question is, what defines a 'family'? And there is no ultimate answer, beyond the trivial one that a 'family' is a set of contrasts. Whether any particular set of contrasts should be considered a family is a subjective decision. The 3rd, 17th, and 42nd analyses that I ever conducted in my life are a set of contrasts, and I could have adjusted my $\alpha$ threshold to insure that the probability of type I errors amongst them was held at 5%, but no one would find this sensical. The question for you is whether you consider your contrasts to be a set in a meaningful sense, and only you can make that judgment. I will offer some standard approaches. Many analysts believe that if a set of contrasts come from the same experiment / data set, they should be treated as a family, and procedures (such as $\alpha$ adjustment) are necessary. Others believe that even when contrasts come from the same experiment, if they are a-priori and orthogonal, special procedures are not required. Both of these positions can be defended. Finally, note also that procedures to control familywise error rates come at a cost--viz. increased type II error rates.

Answer 2

The criterion is that the hypotheses are interdependent in the sense that if one of them breaks then the whole your conclusion or theory breaks. Hence you need a guarantee that if all the tests are significant none of them is significant falsely.

Answer 3

A discussion on researchgate (http://www.researchgate.net/post/Bonferroni-how_is_the_family_of_hypotheses_defined) provided a list of papers, which might help collecting opinions - the papers actually start from the question "when to apply corrections in a multiple testing situation". The papers -all cited often - are:

1) Rothman KJ. No adjustments are needed for multiple comparisons. Epidemiology.1990;1(1):43-6. http://psg-mac43.ucsf.edu/ticr/syllabus/courses/9/2003/02/27/Lecture/readings/Rothman.pdf

2) Perneger TV. What´s wrong with Bonferroni adjustments. BMJ. 1998;316(7139):1236-8.http://static.sdu.dk/mediafiles/D/1/F/%7BD1F06030-8FA7-4EE2-BB7D-60D683B18EAA%7DWhat_s-wrong%20_with_Bonferroni_adjustments.BMJ.1998.pdf

3) Bender R, Lange S. Adjusting for multiple testing- when and how? J Clin Epidemiol. 2001;54:343-9. http://www.rbsd.de/PDF/multiple.pdf

Summary:

1) and 2) focus on "all null hypotheses are true", called the general null hypothesis. It can be more properly rejected (i.e. no alpha-cummulation) if adjustments for multiple comparisons are applied. However, both 1) and 2) oppose, that the general null hypothesis is rarely fully used in the process of scientific research - so the "whole theory breaks" criterion does not automatically apply, when one/some of the null hypotheses in one's data analysis are rejected by chance. 1) adds, that it is naive to think of single null hypotheses, which were (falsely) rejected will never be revisited by the scientific community again.

3) states that once single hypotheses melt in one argument, the adjustments must be done.

From my point of view 1), 2), 3) together just mirror, how carefully we must the "whole theory breaks" criterion. Neither is there a way to just put all null hypotheses in one big sausage - nor a way to rely on the slices of the sausage presented as many single hypotheses. This is, where empirical work really meets working with theory from the domain under research.