4
$\begingroup$

I'm using Evan Miller's sample size calculator as well as his article "How not to run an A/B test".

In my case, I'm using a conversion rate of 6.3 with a minimum detectable effect of 5% (that is, five percentage points so 1.3% - 11.3% so 5% absolute). I have two questions:

  1. The sample size given by the tool and formula is referring to number of conversions , or is it number of exposures?
  2. I get slightly different results comapred to the tool when using the formula in the article $$n = \frac{\sigma ^{2}}{\delta ^{2}}$$ with $$\sigma ^{2} = p*(1-p)$$

The tool gives me a sample size of 407, while the formula gives 378. This difference gets smaller and smaller with smaller effect (at 1% it's 9468 for the tool and 9445 with the formula) - but why are they different?

  1. The power of the test is 80% (by default). If we let the experiment run to the predetermined sample size and no significant effect is found, is it correct to say with 80% probability there is no effect larger than the minimum detectable effect (in this case 5%)?
$\endgroup$
1
$\begingroup$

Sample size is the number of "exposures". An absolute MDE of 5% seems huge vs 6.3% conversion. That's why you need such a small sample. My interpretation of test power is the same (there is 20% chance you won't detect an effect)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.