2
$\begingroup$

I need to run an A-B testing with one control group (40%) and 3 variants (20%) each.

Base line = 0.65

Target lift = 0.25

Power =0.8

Confidence = 95%

I found following R code to find minimum sample size for two variants.

power.prop.test(p1=.65,p2=.65*1.25,power=.8, alternative="one.sided", sig.level = 0.05)

How to calculate sample size needed for A-B testing considering there are 3 variants(60%) and one control group (40%)?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ It depends on what your intended test is: what will be the null hypothesis and the alternative hypothesis? It also depends on how you intend to run the experiment and how you will test the hypothesis. Please supply these details. $\endgroup$
    – whuber ♦
    Commented Jun 18, 2019 at 18:55
  • $\begingroup$ @whuber it will be click through rate. I don't have much knowledge about stats, but the experiment is going to be like-> we will send two templates with some modification and see which one is better. $\endgroup$
    – Vipul Aggarwal
    Commented Jun 18, 2019 at 18:58
  • $\begingroup$ Your response is not helpful (for me anyhow). Here is the result from one computation. You have a single group. The current success rate is 0.65. You try a new method. If the success rate improves to .90 or better you want to detect that with probability .8 using a test with significance level .05. Then you need $n = 18$ subjects. I don't suppose that is what you want, but can you explain more clearly what you do want. // Most software for comparing two groups give results for equal sample sizes in the two groups. Through simulation one can use $\ne$ gp sizes, but that's not best design. $\endgroup$
    – BruceET
    Commented Jun 18, 2019 at 23:48

1 Answer 1

Reset to default
1
$\begingroup$

Being somewhat conservative we can dismiss the presence of three variants B/C/D and treat each control vs variant X comparison independently. This will indeed allow us to use the functionality provided by power.prop.test. Nevertheless, to ensure that these sample size computations are relevant we will need to adjust the level of significance to correct for multiple testing. With three variants B/C/D simply doing a Bonferroni correction is probably adequate (i.e. something like : power.prop.test( ..., sig.level = 0.05/3)) . Witte et al. (2000) On the relative sample size required for multiple comparisons perform a similar approach when examining differences in normally distributed variables. Jung et al. (2005) Sample size calculation for multiple testing in microarray data analysis perform a more in-depth analysis and outline a methodology for a full simulation study if we want to try this with data particular to a specific application.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.