0
$\begingroup$

I have sales data for 500 stores and lets say I want to run an AB test on a sample of test and control stores with the objective of getting statistically significant increase in sales(lift). My question is how do I calculate the number of test stores(sample size) required for getting a statistically significant result.

One approach I thought of was to use the following method:

enter image description here

where Zaplha is the probability of type 1 error

Z beta probability of type 2 error

Delta is the standard deviation

d is the expected change from mean i.e. effect size

My only concern is how do I accurately calculate standard deviation of the population in this case, i.e. taking just an average of standard deviation across all the stores may not be the most correct method?

Any pointers would be appreciated.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ The formula you show is for a two-sample z test, assuming that A and B groups are from populations with the same variance $\sigma^2$ (or in your formula $\delta^2).$ I don't know what you mean by an 'average of the standard deviation across all stores'. Do you somehow have a standard deviation for each store? // You don't say what variable you will be using for the test or what you are trying to find out. Perhaps an example with data for the first few scores in each group would help. $\endgroup$
    – BruceET
    Jun 3, 2020 at 1:27
  • $\begingroup$ The variable I am using is sales for each store and I want to find out that if i make a change in some of the store then does it significantly increase the sales or not, by observing the test vs control sales change $\endgroup$
    – bakas
    Jun 3, 2020 at 2:46
  • 1
    $\begingroup$ Then estimate SD by taking the SD of sales at all 500 stores. Not saying it will be exactly right when you do the test, but seem the best available estimate. // Your formula is for a z test assuming your estimate of $\sigma$ is correct. There are power and sample size procedures in many statistical software programs that can give sample size for a t test (in which SDs are based on samples used for test. \\ Your formula for z test will tend to underestimate $n$ for each group in 2-sample t test. Only slightly, I hope. $\endgroup$
    – BruceET
    Jun 3, 2020 at 5:10
  • $\begingroup$ Thanks BruceET! $\endgroup$
    – bakas
    Jun 3, 2020 at 13:56

1 Answer 1

Reset to default
1
$\begingroup$

You need the variance of only those stores on which you are conducting the test. It is important to understand that A and B should be similar before the experiment (And A test). In general, go from stores to users, conversions (to the target metric that the test is aimed at)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.