9
$\begingroup$

When trying to evaluate what constitutes a family of hypotheses within an experiment/project/analysis, I've found "similar in purpose" and "similar in content" given as guidelines for delimiting families, but these leave quite a lot open to interpretation (to say the least).

It seems clear that if in the course of an analysis, I do several tests of group means and a separate batch of tests of homogeneity of proportions, that I wouldn't bundle everything up together into a single family of hypotheses.

However, if I have several batches of somewhat related tests of group means, what criterion brings them together into a family (or splits them into separate families)? Should all members of a family have the same response variable? If I had different response variables but the same set of cases involved, would those all get bundled into a family of hypotheses?

$\endgroup$
4
$\begingroup$

The issue of multiple comparisons is a really big topic. There have been many opinions and many disagreements. This is due to many things; among others, it is partly because the issue is really important, and partly because there really is no ultimate rule or criterion. Take a prototypical case: You conduct an experiment with $k$ treatments and get a significant ANOVA, so now you wonder which treatment means differ. How should you go about this, run $k(k-1)/2$ t-tests? Although these tests would individually hold $\alpha$ at .05, the 'familywise' $\alpha$ (i.e., the probability that at least 1 type I error will occur) will explode. In fact, the familywise error rate will be $1-(1-\alpha)^k$. The question is, what defines a 'family'? And there is no ultimate answer, beyond the trivial one that a 'family' is a set of contrasts. Whether any particular set of contrasts should be considered a family is a subjective decision. The 3rd, 17th, and 42nd analyses that I ever conducted in my life are a set of contrasts, and I could have adjusted my $\alpha$ threshold to insure that the probability of type I errors amongst them was held at 5%, but no one would find this sensical. The question for you is whether you consider your contrasts to be a set in a meaningful sense, and only you can make that judgment. I will offer some standard approaches. Many analysts believe that if a set of contrasts come from the same experiment / data set, they should be treated as a family, and procedures (such as $\alpha$ adjustment) are necessary. Others believe that even when contrasts come from the same experiment, if they are a-priori and orthogonal, special procedures are not required. Both of these positions can be defended. Finally, note also that procedures to control familywise error rates come at a cost--viz. increased type II error rates.

$\endgroup$
3
$\begingroup$

The criterion is that the hypotheses are interdependent in the sense that if one of them breaks then the whole your conclusion or theory breaks. Hence you need a guarantee that if all the tests are significant none of them is significant falsely.

$\endgroup$
  • $\begingroup$ So, running thousands of t-tests across different measurements in a before-and-after-treatment experiment (like a gene expression experiment) would not count as a family of tests? One false positive wouldn't be desirable, but it wouldn't completely break the conclusions from the experiment as a whole. $\endgroup$ – Ryan Dec 19 '11 at 18:37
  • $\begingroup$ I think so. If that wasn't sound a statistician should wish to die young or quit profession soon, in order to escape multiplying type I error in his life course. $\endgroup$ – ttnphns Dec 19 '11 at 19:36
  • $\begingroup$ Ok, alright. Following strict boolean logic in a world where all problems are like those encountered in casino and other simple games, one type I error certainly would invalidate the entire theory. $\endgroup$ – Ryan Dec 20 '11 at 2:18
2
$\begingroup$

A discussion on researchgate (http://www.researchgate.net/post/Bonferroni-how_is_the_family_of_hypotheses_defined) provided a list of papers, which might help collecting opinions - the papers actually start from the question "when to apply corrections in a multiple testing situation". The papers -all cited often - are:

1) Rothman KJ. No adjustments are needed for multiple comparisons. Epidemiology.1990;1(1):43-6. http://psg-mac43.ucsf.edu/ticr/syllabus/courses/9/2003/02/27/Lecture/readings/Rothman.pdf

2) Perneger TV. What´s wrong with Bonferroni adjustments. BMJ. 1998;316(7139):1236-8.http://static.sdu.dk/mediafiles/D/1/F/%7BD1F06030-8FA7-4EE2-BB7D-60D683B18EAA%7DWhat_s-wrong%20_with_Bonferroni_adjustments.BMJ.1998.pdf

3) Bender R, Lange S. Adjusting for multiple testing- when and how? J Clin Epidemiol. 2001;54:343-9. http://www.rbsd.de/PDF/multiple.pdf

Summary:

1) and 2) focus on "all null hypotheses are true", called the general null hypothesis. It can be more properly rejected (i.e. no alpha-cummulation) if adjustments for multiple comparisons are applied. However, both 1) and 2) oppose, that the general null hypothesis is rarely fully used in the process of scientific research - so the "whole theory breaks" criterion does not automatically apply, when one/some of the null hypotheses in one's data analysis are rejected by chance. 1) adds, that it is naive to think of single null hypotheses, which were (falsely) rejected will never be revisited by the scientific community again.

3) states that once single hypotheses melt in one argument, the adjustments must be done.

From my point of view 1), 2), 3) together just mirror, how carefully we must the "whole theory breaks" criterion. Neither is there a way to just put all null hypotheses in one big sausage - nor a way to rely on the slices of the sausage presented as many single hypotheses. This is, where empirical work really meets working with theory from the domain under research.

$\endgroup$
  • $\begingroup$ These are good papers, but maybe you could give a succinct summary of what they say in relation to this question? Just in case the links go down, we would like to have record. $\endgroup$ – Chris C Sep 29 '15 at 19:48
  • $\begingroup$ Glad to hear, if summaries helped (or not). $\endgroup$ – Statos Oct 20 '15 at 12:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.