I'm running an A/B test in which we have 4 different messages on a landing page (the current one and 3 treatments/variations). My task is to determine the sample size for this test, based on a 5% significance level and 80% power. The metric/variable of interest is conversion (% of users that signed-up on our website). Let each page be p1, p2, p3 p4, p1 being the current one. I have the expected conversion rate for each page (p1 = 20%, p2 = 22%, p3 = 25% and p4 = 30%).

How do I compute the sample size for each landing page (treatment)? One simple approach is just to compute the sample size for a proportion, assuming the null is p1 and the alternative is p4. Within R, it would be: power.prop.test (p1=0.2, p2 = .3, sig.level=0.05, power=0.8)

However, that doesn't seem right, because we have two other pages that I'm testing. Another approach would be to run tests between two pages at a time, pick the winner, and put the winner against another competitor, until the final winner is found. But, I think I'll have a multiple comparisons problem, and also, I already have prior information about the winner on each round of tests (I could use Bayes here, but don't want to go this route. Also, it would take more time to have an answer, and running four experiments in parallel will provide answers faster).

So, how should I compute the sample size in this case? Do I have to look at the false discovery rate literature? Can I use Bonferroni correction when I put the alpha on the power.prop.test function in R? Is this similar to factorial experiment (though not exactly equal) and, if so, how can it help me answer my question?

Any points to the literature will be appreciated.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.