# Sample size calculation for AB testing in offline scenario

I have sales data for 500 stores and lets say I want to run an AB test on a sample of test and control stores with the objective of getting statistically significant increase in sales(lift). My question is how do I calculate the number of test stores(sample size) required for getting a statistically significant result.

One approach I thought of was to use the following method:

where Zaplha is the probability of type 1 error

Z beta probability of type 2 error

Delta is the standard deviation

d is the expected change from mean i.e. effect size

My only concern is how do I accurately calculate standard deviation of the population in this case, i.e. taking just an average of standard deviation across all the stores may not be the most correct method?

Any pointers would be appreciated.

• The formula you show is for a two-sample z test, assuming that A and B groups are from populations with the same variance $\sigma^2$ (or in your formula $\delta^2).$ I don't know what you mean by an 'average of the standard deviation across all stores'. Do you somehow have a standard deviation for each store? // You don't say what variable you will be using for the test or what you are trying to find out. Perhaps an example with data for the first few scores in each group would help. Jun 3, 2020 at 1:27
• The variable I am using is sales for each store and I want to find out that if i make a change in some of the store then does it significantly increase the sales or not, by observing the test vs control sales change Jun 3, 2020 at 2:46
• Then estimate SD by taking the SD of sales at all 500 stores. Not saying it will be exactly right when you do the test, but seem the best available estimate. // Your formula is for a z test assuming your estimate of $\sigma$ is correct. There are power and sample size procedures in many statistical software programs that can give sample size for a t test (in which SDs are based on samples used for test. \\ Your formula for z test will tend to underestimate $n$ for each group in 2-sample t test. Only slightly, I hope. Jun 3, 2020 at 5:10
• Thanks BruceET! Jun 3, 2020 at 13:56