I'm trying to determine what constitutes a family of hypotheses when measuring multiple response variables. In particular I have an A/B type test where I'm comparing several marketing campaigns against a baseline campaign.

For each campaign I want to measure increase in revenue, clicks, and purchases. What I'm not sure of is how many families of hypotheses exist in this situation? Is it more correct to group tests together based on the response variable of interest, or should all the tests be considered one big family?

Concretely, assume I have 4 campaigns, and I want to compare each campaign against the baseline.

Does this mean I have three families of hypothesis, one each for revenue, clicks, and purchases, and each family contains 4 hypothesis? Or is it better to say I have one family of hypothesis, containing 12 individual hypotheses (4 comparisons over three variables)?


If you use a false discovery rate technique to adjust for multiple comparisons, you have the advantage that inferences based on one set of tests will be consistent with inferences based on the original set plus an arbitrary number of new tests. This is in contrast to family-wise error rate adjustments, for which inferences are not stable to enlarging the size of the family of tests. You also gain the advantage of added statistical power with FDR methods.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.