Imagine you're doing an A/B-test in Marketing. You send a control group the standard e-mail, and a test group an e-mail in which one element was changed (for example the color of a button). You want to compare the conversion rates of the e-mails, to determine whether the new e-mail leads to a different conversion rate.
Suppose that the base conversion rate is 16%, and you're interested in a Minimun Detectable Effect of at least 2 percentage-points. Power is set at 80% and the significance level at 5%.
In order to determine the minimum number of recipients per version, I used this calculator:
https://www.evanmiller.org/ab-testing/sample-size.html#!16;80;5;2;0
The suggested sample size per variation is 5,352 recipients per version.
However, if I use the calculator below to determine whether a result is significant, I can find significance at much smaller sample sizes:
https://www.socscistatistics.com/tests/ztest/default2.aspx
For example, if I enter 0.16 and 0.18 (two percentage-points higher) as achieved proportions (conversion rates) and sample sizes of 2,725 for each sample, the result is already significant.
Question
Why would the first calculator suggest a much larger sample size than is required for significance, for the same resulting proportions? (or in more general terms: is it true that the sample size should be much larger than what seems to be minimally required to achieve significance? And if so, why?)
Own guess
My guess is that it has something to do with uncertainty and variability: a larger sample size would make it more likely to find a detectable effect in case of some random noise. I also thought it might have to do with that the first calculator considers a double-sided hypothesis (the effect can go either way). But then in the second calculator, I do select the option for two-tailed test. As you can see I'm not sure about it, and of course these explanations are not very formal and well-formulated. Hence any help would be greatly appreciated.
