Let's say I want to run an experiment where there's a control and there's a feature group (say, 50-50 random split). The treatment can affect the feature group in 1, 2 or three ways (or any combination of the three).

Think, for example, on an online retailer that wants to run an email marketing campaign. The marketing team tries to variants of the email and the primary metric they want to test is if the mean of the total value spent on the website is higher for one of the groups. A secondary hypothesis would be if one of the emails increased the frequency of purchase.

How would you do a sample size calculation for that? Assuming we correct the alpha with Bonferroni, set beta at .2 and know the effect size we want to find in each one, it's straightforward to calculate sample size required per hypotheses. That said, how do you roll up the different three to get the total sample size you need for all hypotheses to reach sample size required? Just take the max of the individual sample sizes?

Thanks for your help!


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.