I'm using Evan Miller's sample size calculator as well as his article "How not to run an A/B test".
In my case, I'm using a conversion rate of 6.3 with a minimum detectable effect of 5% (that is, five percentage points so 1.3% - 11.3% so 5% absolute). I have two questions:
- The sample size given by the tool and formula is referring to number of conversions , or is it number of exposures?
- I get slightly different results comapred to the tool when using the formula in the article $$n = \frac{\sigma ^{2}}{\delta ^{2}}$$ with $$\sigma ^{2} = p*(1-p)$$
The tool gives me a sample size of 407, while the formula gives 378. This difference gets smaller and smaller with smaller effect (at 1% it's 9468 for the tool and 9445 with the formula) - but why are they different?
- The power of the test is 80% (by default). If we let the experiment run to the predetermined sample size and no significant effect is found, is it correct to say with 80% probability there is no effect larger than the minimum detectable effect (in this case 5%)?