2
$\begingroup$

I have 2 new conversion webpages to test. Leadership recommends 80% traffic to existing conversion webpage and 10% each to the 2 new conversion webpages (non-negotiable)

How do I calculate sample size? I understand the fundamentals of sample size calculation - alpha, power, baseline conversion rate and minimum detectable effect needed but have done it for a 50/50 distribution and 1 treatment only..

I am unsure how to calculate for 2 treatment and uneven distribution. Can anyone guide me or refer me to a calculator or formula?

$\endgroup$
1
  • $\begingroup$ What null and alternative? What test statistic? $\endgroup$
    – Glen_b
    May 18, 2023 at 3:46

1 Answer 1

0
$\begingroup$

Main point: use a standard RCT methodology but adjust your $\alpha$ level accordingly to account for multiple comparisons.

More details: Irrespective of how many arms we have and their respective sizes, it is still the case of the smaller sample sizes dominating the power of the study - smaller sample sizes inflate the occurrence of Type II errors. That means we need to run our experiment longer (to get more data) and/or make the different arms of the study have "more dramatic" changes (to get larger expected effect sizes). On top of that, we need to do a correction for multiple comparisons in our $\alpha$ too because we now have two comparisons, i.e. use a family-wise error rate for an adjusted level of $\frac{\alpha}{k}$ is usually fine ($k$=2 here). Notice that this latter point is counter to the usual idea of increasing $\alpha$ to avoid Type II errors. If one wants to explore this behaviour a bit further using R, the function stats::p.adjust is immediately available in every R installation and offers a good place to start.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.