I have sales data for 500 stores and lets say I want to run an AB test on a sample of test and control stores with the objective of getting statistically significant increase in sales(lift). My question is how do I calculate the number of test stores(sample size) required for getting a statistically significant result.
One approach I thought of was to use the following method:
where Zaplha is the probability of type 1 error
Z beta probability of type 2 error
Delta is the standard deviation
d is the expected change from mean i.e. effect size
My only concern is how do I accurately calculate standard deviation of the population in this case, i.e. taking just an average of standard deviation across all the stores may not be the most correct method?
Any pointers would be appreciated.