Sample size for A/B testing

Question

I'm using Evan Miller's sample size calculator as well as his article "How not to run an A/B test".

In my case, I'm using a conversion rate of 6.3 with a minimum detectable effect of 5% (that is, five percentage points so 1.3% - 11.3% so 5% absolute). I have two questions:

1. The sample size given by the tool and formula is referring to number of conversions , or is it number of exposures?
2. I get slightly different results comapred to the tool when using the formula in the article $$n = \frac{\sigma ^{2}}{\delta ^{2}}$$ with $$\sigma ^{2} = p*(1-p)$$

The tool gives me a sample size of 407, while the formula gives 378. This difference gets smaller and smaller with smaller effect (at 1% it's 9468 for the tool and 9445 with the formula) - but why are they different?

3. The power of the test is 80% (by default). If we let the experiment run to the predetermined sample size and no significant effect is found, is it correct to say with 80% probability there is no effect larger than the minimum detectable effect (in this case 5%)?

Answer 1

Sample size is the number of "exposures". An absolute MDE of 5% seems huge vs 6.3% conversion. That's why you need such a small sample. My interpretation of test power is the same (there is 20% chance you won't detect an effect)