Could someone help point me to what statistical method I should be using to calculate the desired sample size?

We are currently in the process of implementing a new change, our hypothesis is that this change would lead to a 10% lift (relative change) in the conversion rate from 20% (current baseline) to 22%(after implementation of change). I want to understand what is the best way to calculate the sample size and how long we need to measure after the change until we can conclude the lift is statistically significant.


1 Answer 1


Here is a rough idea. For large $n,$ an approximate 95% confidence interval for the true conversion rate $p$ is of the form $$\hat p \pm 1.96\sqrt{\hat p(1-\hat p)/n},$$ where $\hat p$ is the estimated conversion rate.

It seems you want to detect a difference of $2\% = 0.02.$ Setting the margin or error $1.96\sqrt{\hat p(1-\hat p)/n} = 0.02$ leads to approximately $n = 1600.$

For a more precise computation you could set up a test of hypothesis, specify the significance level and desired power for detecting a difference of 0.02, and use one of many 'power and sample size' utilities that provide the required value of $n.$

  • $\begingroup$ Thank you BruceET! I came across a formula provided in another post used to calculate the sample size, would this be an appropriate method for my example above? 𝑛=(𝑍α/2+𝑍β)2βˆ—(𝑝1(1βˆ’π‘1)+𝑝2(1βˆ’π‘2))/(𝑝1βˆ’π‘2)^2 And another follow up question I have is: in general if a difference in the conversion rate detected, to see if this is statistically significant against the historical conversion rate. What is the most appropriate statistical test I should use to do this? Is a chi square test appropriate here? $\endgroup$
    – StealthEXE
    Commented Feb 25, 2020 at 3:27
  • $\begingroup$ The link the post is here: stats.stackexchange.com/questions/178568/… $\endgroup$
    – StealthEXE
    Commented Feb 25, 2020 at 3:30

Your Answer

By clicking β€œPost Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.